014.Docker Harbor+Keepalived+LVS+Shared Storage High Availability Architecture

More than one introduction to Harbor High Availability

Shared back-end storage is a standard solution for sharing multiple Harbor instances with the same back-end storage, and any instance that persists to a stored image can be read by other instances.Pre-LB components, such as Keepalived, can be shunted to different instances for processing, thus achieving load balancing and avoiding single point failures. The architecture diagram is as follows:
Program description:
Shared storage: Harbor's back-end storage currently supports AWS S3, Openstack Swift, Ceph, etc. This experimental environment uses NFS;
Shared Session:harbor default sessions are stored in redis, so redis can be separated to share sessions on different instances. Independent redis can also use redis sentinel or redis cluster to ensure high availability of redis. This experimental environment uses a single redis;
Database highly available: MySQL multiple instances cannot share a single MySQL data file and can isolate the database in harbor.Let multiple instances share an external database. Independent MySQL databases can also use mysqls cluster to ensure the high availability of mysql. This experimental environment uses a single mysql.

Second Official Deployment

2.1 Preparations

 
node
IP Address
Remarks
docker01
172.24.8.111
Docker harbor node01
docker02
172.24.8.112
Docker harbor node02
docker03
172.24.8.113
mysql+redis node
docker04
172.24.8.114
Docker client, for testing warehouses
nfsslb
172.24.8.71
Shared nfs storage node
Keepalived node
VIP address: 172.24.8.200/32
slb02
172.24.8.72
Keepalived node
VIP address: 172.24.8.200/32
 
Schematic architecture:
Pre-configuration:
  • Installation of docker, docker-compose (see Basic Use of 009.Docker Compose);
  • ntp clock synchronization (recommended);
  • Relevant firewall-SELinux is turned on or off;
  • nfsslb and slb02 node add resolution: echo "172.24.8.200 reg.harbor.com" >/etc/hosts

2.2 Create nfs

  1 [root@nfsslb ~]# yum -y install nfs-utils*
  2 [root@nfsslb ~]# mkdir /myimages		#For sharing mirrors
  3 [root@nfsslb ~]# mkdir /mydatabase		#Used to store database data
  4 [root@nfsslb ~]# echo -e "/dev/vg01/lv01 /myimages ext4 defaults 0 0\n/dev/vg01/lv02 /mydatabase ext4 defaults 0 0">> /etc/fstab
  5 [root@nfsslb ~]# mount -a
  6 [root@nfsslb ~]# vi /etc/exports
  7 /myimages 172.24.8.0/24(rw,no_root_squash)
  8 /mydatabase 172.24.8.0/24(rw,no_root_squash)
  9 [root@nfsslb ~]# systemctl start nfs.service
 10 [root@nfsslb ~]# systemctl enable nfs.service
 
Note: The nfsserver node uses a stand-alone LVM disk as the NFS mount directory and configures the corresponding shared directory. For more NFS configurations, see NFS 004.NFS Configuration Instance.

2.3 Mount nfs

  1 root@docker01:~# apt-get -y install nfs-common
  2 root@docker02:~# apt-get -y install nfs-common
  3 root@docker03:~# apt-get -y install nfs-common
  4 
  5 root@docker01:~# mkdir /data
  6 root@docker02:~# mkdir /data
  7 
  8 root@docker01:~# echo "172.24.8.71:/myimages /data nfs defaults,_netdev 0 0">> /etc/fstab
  9 root@docker02:~# echo "172.24.8.71:/myimages /data nfs defaults,_netdev 0 0">> /etc/fstab
 10 root@docker03:~# echo "172.24.8.71:/mydatabase /database nfs defaults,_netdev 0 0">> /etc/fstab
 11 
 12 root@docker01:~# mount -a
 13 root@docker02:~# mount -a
 14 root@docker03:~# mount -a
 15 
 16 root@docker03:~# mkdir -p /database/mysql
 17 root@docker03:~# mkdir -p /database/redis
 

2.4 Deploy external mysql-redis

  1 root@docker03:~# mkdir docker_compose/
  2 root@docker03:~# cd docker_compose/
  3 root@docker03:~/docker_compose# vi docker-compose.yml
  4 version: '3'
  5 services:
  6   mysql-server:
  7     hostname: mysql-server
  8     restart: always
  9     container_name: mysql-server
 10     image: mysql:5.7
 11     volumes:
 12       - /database/mysql:/var/lib/mysql
 13     command: --character-set-server=utf8
 14     ports:
 15       - '3306:3306'
 16     environment:
 17       MYSQL_ROOT_PASSWORD: x19901123
 18 #    logging:
 19 #      driver: "syslog"
 20 #      options:
 21 #        syslog-address: "tcp://172.24.8.112:1514"
 22 #        tag: "mysql"
 23   redis:
 24     hostname: redis-server
 25     container_name: redis-server
 26     restart: always
 27     image: redis:3
 28     volumes:
 29       - /database/redis:/data
 30     ports:
 31       - '6379:6379'
 32 #    logging:
 33 #      driver: "syslog"
 34 #      options:
 35 #        syslog-address: "tcp://172.24.8.112:1514"
 36 #        tag: "redis"
 
Tip: Because the log container serves in harbor, when harbor is not deployed yet, you need to comment on the configuration. When harbor is deployed, uncomment it and then up grade it again.
  1 root@docker03:~/docker_compose# docker-compose up -d
  2 root@docker03:~/docker_compose# docker-compose ps		#Confirm docker up
  3 root@docker03:~/docker_compose# netstat -tlunp			#Verify whether related ports are started
 

2.5 Download harbor

  1 root@docker01:~# wget https://storage.googleapis.com/harbor-releases/harbor-offline-installer-v1.5.4.tgz
  2 root@docker01:~# tar xvf harbor-offline-installer-v1.5.4.tgz
 
Tip: The docker02 node reference is as above.

2.6 Importing the registry table

  1 root@docker01:~# apt-get -y install mysql-client
  2 root@docker01:~# cd harbor/ha/
  3 root@docker01:~/harbor/ha# ll
 
  1 root@docker01:~/harbor/ha# mysql -h172.24.8.113 -uroot -p
  2 mysql> set session sql_mode='STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';					#sql_mode must be modified
  3 mysql> source ./registry.sql			#Import the registry data table to the external database.
  4 mysql> exit
 
Tip: Only import once.

2.7 Modify harbor-related configuration

  1 root@docker01:~/harbor/ha# cd /root/harbor/
  2 root@docker01:~/harbor# vi harbor.cfg				#Modify harbor profile
  3 hostname = 172.24.8.111
  4 db_host = 172.24.8.113
  5 db_password = x19901123
  6 db_port = 3306
  7 db_user = root
  8 redis_url = 172.24.8.113:6379
  9 root@docker01:~/harbor# vi prepare
 10 empty_subj = "/C=/ST=/L=/O=/CN=/"
 11 Modify as follows:
 12 empty_subj = "/C=US/ST=California/L=Palo Alto/O=VMware, Inc./OU=Harbor/CN=notarysigner"
 13 root@docker01:~/harbor# ./prepare				#Load related configurations
 
Tip: docker02 reference can be configured as above;
Because of the use of external mysql and redis, the following schema diagrams show that database-related components have UI and jobservices and need to be modified accordingly. Running the prepare command will automatically synchronize the corresponding database parameters to. /common/config/ui/env and. /common/config/adminserver/env.
  1 root@docker01:~/harbor# cat ./common/config/ui/env		#Verification
  2 _REDIS_URL=172.24.8.113:6379
  3 root@docker01:~/harbor# cat ./common/config/adminserver/env | grep MYSQL	#Verification
  4 MYSQL_HOST=172.24.8.113
  5 MYSQL_PORT=3306
  6 MYSQL_USR=root
  7 MYSQL_PWD=x19901123
  8 MYSQL_DATABASE=registry
 

2.8 docker-compose deployment

  1 root@docker01:~/harbor# cp docker-compose.yml docker-compose.yml.bak
  2 root@docker01:~/harbor# cp ha/docker-compose.yml .
  3 root@docker01:~/harbor# vi docker-compose.yml
  4   log
  5     ports:
  6       - 1514:10514		#log needs to serve external redis and mysql, so you just need to modify it here
  7 root@docker01:~/harbor# ./install.sh
 
Tip: Since redis and mysql are deployed externally, service items for redis and mysql need to be deleted or commented on in docker-compose.yml, and dependencies of other services need to be removed. Modified docker-compose files already exist in the official harbor in the ha directory.
The docker02 node refers to the 2.5-2.8 deployment of harbor.

2.9 Rebuild External redis and mysql

Remove log comment entries.
  1 root@docker03:~/docker_compose# docker-compose up -d
  2 root@docker03:~/docker_compose# docker-compose ps		#Confirm docker up
  3 root@docker03:~/docker_compose# netstat -tlunp			#Verify whether related ports are started
 

2.10 Keepalived Installation

  1 [root@nfsslb ~]# yum -y install gcc gcc-c++ make kernel-devel kernel-tools kernel-tools-libs kernel libnl libnl-devel libnfnetlink-devel openssl-devel
  2 [root@nfsslb ~]# cd /tmp/
  3 [root@nfsslb ~]# tar -zxvf keepalived-2.0.8.tar.gz
  4 [root@nfsslb tmp]# cd keepalived-2.0.8/
  5 [root@nfsslb keepalived-2.0.8]# ./configure --sysconf=/etc --prefix=/usr/local/keepalived
  6 [root@nfsslb keepalived-2.0.8]# make && make install
 
Tip: The slb02 node reference is as above.

2.11 Keepalived configuration

  1 [root@nfsslb ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
  2 root@docker01:~# scp harbor/ha/sample/active_active/keepalived_active_active.conf root@172.24.8.71:/etc/keepalived/keepalived.conf
  3 root@docker01:~# scp harbor/ha/sample/active_active/check.sh root@172.24.8.71:/usr/local/bin/check.sh
  4 root@docker01:~# scp harbor/ha/sample/active_active/check.sh root@172.24.8.72:/usr/local/bin/check.sh
  5 [root@nfsslb ~]# chmod u+x /usr/local/bin/check.sh
  6 [root@slb02 ~]# chmod u+x /usr/local/bin/check.sh
  7 [root@nfsslb ~]# vi /etc/keepalived/keepalived.conf
  8 global_defs {
  9   router_id haborlb
 10 }
 11 vrrp_sync_groups VG1 {
 12   group {
 13     VI_1
 14   }
 15 }
 16 vrrp_instance VI_1 {
 17   interface eth0
 18 
 19   track_interface {
 20     eth0
 21   }
 22 
 23   state MASTER
 24   virtual_router_id 51
 25   priority 10
 26 
 27   virtual_ipaddress {
 28     172.24.8.200
 29   }
 30   advert_int 1
 31   authentication {
 32     auth_type PASS
 33     auth_pass d0cker
 34   }
 35 
 36 }
 37 virtual_server 172.24.8.200 80 {
 38   delay_loop 15
 39   lb_algo rr
 40   lb_kind DR
 41   protocol TCP
 42   nat_mask 255.255.255.0
 43   persistence_timeout 10
 44 
 45   real_server 172.24.8.111 80 {
 46     weight 10
 47     MISC_CHECK {
 48         misc_path "/usr/local/bin/check.sh 172.24.8.111"
 49         misc_timeout 5
 50     }
 51   }
 52 
 53   real_server 172.24.8.112 80 {
 54     weight 10
 55     MISC_CHECK {
 56         misc_path "/usr/local/bin/check.sh 172.24.8.112"
 57         misc_timeout 5
 58     }
 59   }
 60 }
 61 [root@nfsslb ~]# scp /etc/keepalived/keepalived.conf root@172.24.8.72:/etc/keepalived/keepalived.conf	#Copy Keepalived configuration to slb02 node
 62 [root@nfsslb ~]# vi /etc/keepalived/keepalived.conf
 63 state BACKUP
 64 priority 8
 
Tip: The official harbor has prompted Keepalived profile and detection script to use directly;
The lsb02 node is set to BACKUP, with a lower priority than MASTER and other defaults.

2.12 slb Node Configuration LVS

  1 [root@nfsslb ~]# yum -y install ipvsadm
  2 [root@nfsslb ~]# vi ipvsadm.sh
  3 #!/bin/sh
  4 #****************************************************************#
  5 # ScriptName: ipvsadm.sh
  6 # Author: xhy
  7 # Create Date: 2018-10-28 02:40
  8 # Modify Author: xhy
  9 # Modify Date: 2018-10-28 02:40
 10 # Version:
 11 #***************************************************************#
 12 sudo ifconfig eth0:0 172.24.8.200 broadcast 172.24.8.200 netmask 255.255.255.255 up
 13 sudo route add -host 172.24.8.200 dev eth0:0
 14 sudo echo "1" > /proc/sys/net/ipv4/ip_forward
 15 sudo ipvsadm -C
 16 sudo ipvsadm -A -t 172.24.8.200:80 -s rr
 17 sudo ipvsadm -a -t 172.24.8.200:80 -r 172.24.8.111:80 -g
 18 sudo ipvsadm -a -t 172.24.8.200:80 -r 172.24.8.112:80 -g
 19 sudo ipvsadm
 20 sudo sysctl -p
 21 [root@nfsslb ~]# chmod u+x ipvsadm.sh
 22 [root@nfsslb ~]# echo "source /root/ipvsadm.sh" >> /etc/rc.local	#Start up and run
 23 [root@nfsslb ~]# ./ipvsadm.sh
 
Example explanation:
ipvsadm -A -t 172.24.8.200:80 -s rr -p 600
Represents the addition of a virtual server with IP 192.168.10.200 to the list of virtual servers in the kernel, and specifies that the service port of the virtual server is 80, its scheduling policy is in polling mode, and the duration on each Real Server is 600 seconds.
ipvsadm -a -t 172.24.8.200:80 -r 192.168.10.100:80 -g
Indicates that a new Real Server record has been added to the virtual server with IP status 192.168.10.10, and the virtual server works in direct routing mode.
Tip: slb02 node can refer to the above configuration, more LVS can refer to https://www.cnblogs.com/itzgr/category/1367969.html.

2.13 harbor Node Configuration VIP

  1 root@docker01:~# vi /etc/init.d/lvsrs
  2 #!/bin/bash
  3 # description:Script to start LVS DR real server.
  4 #
  5 . /etc/rc.d/init.d/functions
  6 VIP=172.24.8.200
  7 #Modify the corresponding VIP
  8 case "$1" in
  9 start)
 10     #Start LVS-DR mode, real server on this machine.Turn off ARP conflict detection.
 11     echo "Start LVS of Real Server!"
 12     /sbin/ifconfig lo down
 13     /sbin/ifconfig lo up
 14     echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
 15     echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
 16     echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
 17     echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
 18     /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
 19     /sbin/route add -host $VIP dev lo:0
 20     sudo sysctl -p
 21 ;;
 22 stop)
 23     #Stop LVS-DR real server loopback device(s).
 24     echo "Close LVS Director Server!"
 25     /sbin/ifconfig lo:0 down
 26     echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
 27     echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
 28     echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
 29     echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
 30     sudo sysctl -p
 31 ;;
 32 status)
 33     # Status of LVS-DR real server.
 34     islothere=`/sbin/ifconfig lo:0 | grep $VIP`
 35     isrothere=`netstat -rn | grep "lo:0" | grep $VIP`
 36     if [ ! "$islothere" -o ! "isrothere" ];then
 37     # Either the route or the lo:0 device
 38     # not found.
 39         echo "LVS-DR real server Stopped!"
 40     else
 41         echo "LVS-DR real server Running..."
 42     fi
 43 ;;
 44 *)
 45     # Invalid entry.
 46     echo "$0: Usage: $0 {start|status|stop}"
 47     exit 1
 48 ;;
 49 esac
 50 root@docker01:~# chmod u+x /etc/init.d/lvsrs
 51 root@docker02:~# chmod u+x /etc/init.d/lvsrs
 

2.14 Start related services

  1 root@docker01:~# service lvsrs start
  2 root@docker02:~# service lvsrs start
  3 [root@nfsslb ~]# systemctl start keepalived.service
  4 [root@nfsslb ~]# systemctl enable keepalived.service
  5 [root@slb02 ~]# systemctl start keepalived.service
  6 [root@slb02 ~]# systemctl enable keepalived.service
 

2.15 Confirmation Verification

  1 root@docker01:~# ip addr			#Verify docker01/02/slb successfully enabled vip

Three test validation

  1 root@docker04:~# vi /etc/hosts
  2 172.24.8.200 reg.harbor.com
  3 root@docker04:~# vi /etc/docker/daemon.json
  4 {
  5    "insecure-registries": ["http://reg.harbor.com"]
  6 }
  7 root@docker04:~# systemctl restart docker.service
  8 If Trust CA Certificates issued by institutions, closed accordingly daemon.json Configuration in.
  9 root@docker04:~# docker login reg.harbor.com		#Log on registry
 10 Username: admin
 11 Password: Harbor12345
 
Tip: Public registries can be pulled, but push must also be logged in, and private registries must be logged in to pull and push.
  1 root@docker04:~# docker pull hello-world
  2 root@docker04:~# docker tag hello-world:latest reg.harbor.com/library/hello-world:xhy
  3 root@docker04:~# docker push reg.harbor.com/library/hello-world:xhy
 
Tip: Modifying tag must be an existing item with appropriate authorization.
Browser access: https://reg.harbor.com and use the default user name admin/Harbor12345
Reference link: https://www.cnblogs.com/breezey/p/9444231.html

Origin: https://www.cnblogs.com/itzgr/p/10166760.html

Tags: Docker MySQL Redis Database

Posted on Fri, 17 Apr 2020 11:51:37 -0700 by jbingman