[Centos] deploy the helpingscheduler from scratch

1. Environment configuration

Install JAVA 1.8 / MySQL 5.7 / ZooKeeper 3.5.8

1.1 installing JAVA 1.8
  • Check whether there is an installation. If there is one, delete it first
rpm -qa | grep java  # java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el7_0.x86_64
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el7_0.x86_64
rpm -ivh jdk-8u231-linux-x64.rpm
  • Add environment variables in the file / etc/profile
export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH
  • Reload profile
source /etc/profile
1.2 install MySQL 5.7

Please refer to this article: [Centos] install and set up telnet MySQL 5.7 (8.0)

1.3 installation of ZooKeeper 3.5.8
wget https://downloads.apache.org/zookeeper/stable/apache-zookeeper-3.5.8.tar.gz
tar -zxf apache-zookeeper-3.5.8.tar.gz
  • to configure
cd apache-zookeeper-3.5.8
cp conf/zoo_sample.cfg conf/zoo.cf

If it is a cluster, it needs to be in conf/zoo.cf It can be set in the file without modification during stand-alone deployment

  • start-up
bin/zkServer.sh start

2. Backend deployment dolphin scheduler

2.1 download
mkdir -p /opt/dolphinscheduler;
cd /opt/dolphinscheduler;
wget https://mirrors.tuna.tsinghua.edu.cn/apache/incubator/dolphinscheduler/1.2.0/apache-dolphinscheduler-incubating-1.2.0-dolphinscheduler-backend-bin.tar.gz
# decompression 
tar -zxvf apache-dolphinscheduler-incubating-1.2.0-dolphinscheduler-backend-bin.tar.gz -C /opt/dolphinscheduler;
# rename
mv apache-dolphinscheduler-incubating-1.2.0-dolphinscheduler-backend-bin  dolphinscheduler-backend
2.2 create new user and authorize
# User creation requires root login
useradd dolphinscheduler;
# Add password
echo "dolphinscheduler" | passwd --stdin dolphinscheduler
# Configure sudo password free
sed -i '$adolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' /etc/sudoers
sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers
# Modify the directory permissions so that the deployment user has operation permissions on the helpingscheduler backend directory  
chown -R dolphinscheduler:dolphinscheduler dolphinscheduler-backend
2.3 ssh password free login configuration
su dolphinscheduler;

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
2.4 database initialization
  • Download JDBC 5.1
cd /opt
wget https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-java-5.1.48.tar.gz
tar zxf mysql-connector-java-5.1.48.tar.gz 
cp /opt/mysql-connector-java-5.1.48/mysql-connector-java-5.1.48.jar /opt/dolphinscheduler/dolphinscheduler-backend/lib
  • Build database
    The user and password are customized here. The next step is to configure the connection
CREATE USER "user"@"%" IDENTIFIED BY "password";
CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}'
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}'
flush privileges
  • Configure database connection
cd /opt/dolphinscheduler/dolphinscheduler-backend
nano conf/application-dao.properties 

There are three areas in the configuration file that need to be modified:

  1. Note configuration of postgre
  2. Remove the comment of mysql
  3. Write ip/username/password parameter
  • Database table building
sh script/create-dolphinscheduler.sh

The creation helpingscheduler success indicates that the database initialization is successful

2.5 start backend service
  • Modify operation parameters
nano /opt/dolphinscheduler/dolphinscheduler-backend/conf/env/.dolphinscheduler_env.sh

Only Java needs to be set in stand-alone mode_ HOME

  • Modify one click deployment script install.sh Parameters in
# Here is mysql or postgresql
dbtype="mysql"
# Database connection address
dbhost="localhost:3306"
# Database name
dbname="dolphinscheduler"
# Database user name, which needs to be modified to the specific value of {user} set above
username="xxx"    
# Database password. If there are special characters, please use \ escape. You need to change to the specific value of {passowrd} set above
passowrd="xxx"
#The directory where DS is installed, such as: / opt / soft / helpingscheduler, is different from the current directory
installPath="/opt/soft/dolphinscheduler"
#Which user to use for deployment? Use the user created in Section 1.3
deployUser="dolphinscheduler"
#Zookeeper address, stand-alone localhost:2181 Remember to bring port 2181
zkQuorum="localhost:2181"
#On which machines to deploy DS service, choose localhost
ips="localhost"
#Which machine is the master service deployed on
masters="localhost"
#Which machine is the worker service deployed on
workers="localhost"
#Which machine is the alarm service deployed on
alertServer="localhost"
#Which machine is the backend api service deployed on
apiServers="localhost"
# Mail configuration, taking qq mailbox as an example
# Mail protocol
mailProtocol="SMTP"
# Mail service address
mailServerHost="smtp.exmail.qq.com"
# Mail service port
mailServerPort="25"
# mailSender and mailUser can be configured as the same
# sender 
mailSender="xxx@qq.com"
# Send user
mailUser="xxx@qq.com"
# Email password
mailPassword="xxx"
# The mailbox of TLS protocol is set to true, otherwise it is set to false
starttlsEnable="true"
# For mail service address value, refer to mailServerHost above
sslTrust="smtp.exmail.qq.com"
# The mailbox configuration to enable SSL protocol is true, otherwise it is false. Note: starttlsEnable and ssnenable cannot be true at the same time
sslEnable="false"
# excel download path
xlsFilePath="/tmp/xls"
# Where to upload resource files such as sql for business, you can set: HDFS,S3,NONE. If you want to use the local file system for a single machine, please configure it as HDFS, because HDFS supports the local file system; if you do not need the resource upload function, please select NONE. One important point: using local file system does not need to deploy hadoop 
resUploadStartupType="HDFS"
# For example, save to local file system
#Note: however, if you want to upload to HDFS and NameNode has HA enabled, you need to set the core-site.xml And hdfs-site.xml Put it in the conf directory. In this example, put it under / opt / helpingscheduler / conf and configure the namenode cluster name. If the NameNode is not HA, change it to a specific ip or host name 
defaultFS="file:///data/dolphinscheduler"    #hdfs: / {specific ip / hostname}: 8020
# *** xx.xx ,192.168. xx.xx ***
yarnHaIps=""
# If it is a single ResourceManager, configure it as the ResourceManager node ip or host name, otherwise keep the default value. I don't use yarn here. Keep it default
singleYarnIp="ark1"
# Because hdfs supports the local file system, it is necessary to ensure that the local folder exists and has read-write permission
hdfsPath="/data/dolphinscheduler"
  • Install the Zookeeper tool kazoo of python
sudo yum -y install python-pip
sudo python3 -m pip install --upgrade pip
sudo pip3 install kazoo
  • Deploy and start services
cd /opt/dolphinscheduler/dolphinscheduler-backend
bash install.sh
  • Stop and start
bash /opt/soft/dolphinscheduler/bin/stop-all.sh
bash /opt/soft/dolphinscheduler/bin/start-all.sh

3. Front end deployment of helpingscheduler

3.1 download
cd /opt/dolphinscheduler
wget https://mirrors.tuna.tsinghua.edu.cn/apache/incubator/dolphinscheduler/1.2.0/apache-dolphinscheduler-incubating-1.2.0-dolphinscheduler-front-bin.tar.gz
tar -zxvf apache-dolphinscheduler-incubating-1.2.0-dolphinscheduler-front-bin.tar.gz -C /opt/dolphinscheduler
mv apache-dolphinscheduler-incubating-1.2.0-dolphinscheduler-front-bin /opt/soft/dolphinscheduler/ui
3.2 Automated Deployment
cd /opt/soft/dolphinscheduler/ui
sh ./install-dolphinscheduler-ui.sh

After execution, you need to type once

  • Front end port: the default port is 8888. If you choose the default port, please enter directly or type another port
  • ip of API server interacting with front-end ui
  • port of API server interacting with front-end ui
  • Operating system selection
3.3 setting up remote access

If the service is running on the server and you want remote access, you need to set nginx.conf file

nano /etc/nginx/conf.d/dolphinscheduler.conf

Modify server_ Name, root and proxy_pass three parameters

server {
    listen       8888;# access port
    server_name  10.0.1.106;
    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;
    location / {
    root   /opt/soft/dolphinscheduler/ui/dist; # static file directory
    index  index.html index.html;
    }
    location /dolphinscheduler {
    proxy_pass http://10.0.1.106:12345; # interface address
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header x_real_ipP $remote_addr;
    proxy_set_header remote_addr $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_connect_timeout 300s;
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection upgrade;
    }
    #error_page  404              /404.html;
    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
    root   /usr/share/nginx/html;
    }
}

Tags: Apache Java MySQL Database

Posted on Tue, 02 Jun 2020 19:45:11 -0700 by dsjoes