Linux -- Shell multi machine distributing files and executing commands

It's a very practical tool.

First, you need to set up ssh password free communication between machines.

There are three documents
1.deploy.conf
2.deploy.sh
3.runRemoteCmd.sh

Be careful:
The three files are all in / home/hadoop/tools Directory;
The first column of the deploy.conf configuration file is the host name of the server, followed by its identification;

Where deploy.conf is the configuration file:

#### NOTES
# There is crontab job using this config file which would compact log files and remove old log file.
# please be  carefully while modifying this file until you know what crontab exactly do
#hdp
cdhnode1,all,master,zk,
cdhnode2,all,slave,zk,
cdhnode3,all,slave,zk,jn,kfk,
cdhnode4,all,slave,jn,kfk,
cdhnode5,all,slave,jn,kfk,

deploy.sh is a multi machine distribution script:

#!/bin/bash
#set -x

if [ $# -lt 3 ]
then
  echo "Usage: ./deply.sh srcFile(or Dir) descFile(or Dir) MachineTag"
  echo "Usage: ./deply.sh srcFile(or Dir) descFile(or Dir) MachineTag confFile"
  exit
fi

src=$1
dest=$2
tag=$3
if [ 'a'$4'a' == 'aa' ]
then
  confFile=/home/hadoop/tools/deploy.conf
else
  confFile=$4
fi

if [ -f $confFile ]
then
  if [ -f $src ]
  then
    for server in `cat $confFile|grep -v '^#'|grep ','$tag','|awk -F',' '{print $1}'`
    do
       scp $src $server":"${dest}
    done
  elif [ -d $src ]
  then
    for server in `cat $confFile|grep -v '^#'|grep ','$tag','|awk -F',' '{print $1}'`
    do
       scp -r $src $server":"${dest}
    done
  else
      echo "Error: No source file exist"
  fi

else
  echo "Error: Please assign config file or run deploy.sh command with deploy.conf in same directory"
fi

runRemoteCmd.sh is a multi machine execution command script:

#!/bin/bash
#set -x

if [ $# -lt 2 ]
then
  echo "Usage: ./runRemoteCmd.sh Command MachineTag"
  echo "Usage: ./runRemoteCmd.sh Command MachineTag confFile"
  exit
fi

cmd=$1
tag=$2
if [ 'a'$3'a' == 'aa' ]
then

  confFile=/home/hadoop/tools/deploy.conf
else
  confFile=$3
fi

if [ -f $confFile ]
then
    for server in `cat $confFile|grep -v '^#'|grep ','$tag','|awk -F',' '{print $1}'`
    do
       echo "*******************$server***************************"
       ssh $server "source ~/.bash_profile; $cmd"
    done
else
  echo "Error: Please assign config file or run deploy.sh command with deploy.conf in same directory"
fi

usage method:

##Distribute the hadoop package to the directory / home/hadoop/app / identified as the slave server:
deploy.sh hadoop /home/hadoop/app/ slave

##Have all servers identified as zk execute the / home/hadoop/app/zookeeper/bin/zkServer.sh start command
runRemoteCmd.sh "/home/hadoop/app/zookeeper/bin/zkServer.sh start" zk

 

Tags: Hadoop ssh crontab Zookeeper

Posted on Mon, 03 Feb 2020 08:00:43 -0800 by jd023