Big data infrastructure

Big data - basic environment construction (1)

server setting

This paper uses three Linux servers as a unified environment.

IP settings for three machines

Modify the ip address of three services

vi /etc/sysconfig/network-scripts/ifcfg-ens33
BOOTPROTO="static"
IPADDR=192.168.52.100
NETMASK=255.255.255.0
GATEWAY=192.168.52.1
DNS1=8.8.8.8

Set the IP addresses of the prepared three servers as follows:
IP address of the first machine: 192.168.52.100
IP address of the second machine: 192.168.52.110
IP address of the third machine: 192.168.52.120

Three machines turn off the firewall

Use the root user to log in to three servers and execute the following command to shut down the firewall

systemctl stop firewalld
systemctl disable firewalld

Three machines shut down selinux

Use the root user to log in to three servers and execute the following command to shut down selinux

vim /etc/selinux/config
SELINUX=disabled

Three machines change the host name

Three machines change the host name respectively
The first host name is changed to: node01.kaikeba.com
The second hostname is changed to: node02.kaikeba.com
The third host name is changed to: node03.kaikeba.com


The first machine (192.168.52.100) executes the following command to modify the hostname

vim /etc/hostname
node01.kaikeba.com

The second machine (192.168.52.110) performs the following command to modify the hostname

vim /etc/hostname
node02.kaikeba.com

The third machine (192.168.52.120) executes the following command to modify the hostname

vim /etc/hostname
node03.kaikeba.com

Three machines change the host name and IP address mapping

Three machines execute the following command to change the host name and IP address mapping relationship

vim /etc/hosts
192.168.52.100 node01.kaikeba.com node01
192.168.52.110 node02.kaikeba.com node02
192.168.52.120 node03.kaikeba.com node03

Synchronization time of three machines

Three machines execute the following commands to synchronize Alibaba cloud server time regularly

yum -y install ntpdate
crontab -e
*/1 * * * * /usr/sbin/ntpdate time1.aliyun.com

Three machines add ordinary users

Three linux servers uniformly add hadoop for ordinary users and give sudo permission for the installation of all big data software in the future
And set the password of ordinary user as hadoop

useradd hadoop
passwd hadoop

Three machines add sudo permission for ordinary users

vi sudo
hadoop ALL=(ALL) ALL

Unified directory of three definitions

Define the storage directory of three linux server software compression packages and the installation directory after decompression. Three machines execute the following command to create two files
The folder is used to store the directory of the software compression package and the directory after decompression.

mkdir -p /kkb/soft # Storage directory of software compression package
mkdir -p /kkb/install # Storage directory after software decompression
chown -R hadoop:hadoop /kkb # Change folder permissions to hadoop users

Three machines to install jdk

Use the hadoop user to reconnect the three machines, and then use the hadoop user to install the jdk software.

Upload the compressed package to / kkb/soft of the first server, then decompress it, configure the environment variables, and install the three machines in turn.
 

cd /kkb/soft/

tar -zxf jdk-8u141-linux-x64.tar.gz -C /kkb/install/

sudo vim /etc/profile

#Add the following configuration content to configure the jdk environment variable
export JAVA_HOME=/kkb/install/jdk1.8.0_141
export PATH=:$JAVA_HOME/bin:$PATH

Password free login for hadoop users

Three machines execute the following commands under hadoop user to generate public key and private key

ssh-keygen -t rsa

#Under hadoop users, three machines execute the following command to copy the public key to node01 server
ssh-copy-id node01

#node01 under hadoop user, execute the following command to copy authorized_keys to node02 and node03 servers
cd /home/hadoop/.ssh/
scp authorized_keys node02:$PWD
scp authorized_keys node03:$PWD

Shutdown and restart of three machines

Three machines execute the following commands under the root user to shut down and restart

reoot -h now

 

75 original articles published, 97 praised, 10000 visitors+
Private letter follow

Tags: Hadoop vim Linux SELinux

Posted on Sun, 09 Feb 2020 23:35:47 -0800 by drisate