Note: In actual production and development, fully distributed is used
1. Construction steps
1) Prepare 3 clients (close firewall, static ip, host name)
2) Install JDK
3) Configure environment Variables
4) Install Hadoop
5) Configure environment variables
6) Configure cluster
7) Single point start
8) Configure ssh
9) Group and test the cluster
2, scp:(secure cope) secure copy
< h4>1, definition of scp:
Scp can realize data copy between server and server. (From server1 to server2)
2, basic syntax
scp -r pdir/pdir/pdir/fname [email protected]@[email protected] :pdir/pdir/pdir/fname
command recursive file path/name to be copied destination user@host: destination path/name
3, case practice
h4>
On hadoop1, copy the software in the /opt/module directory of hadoop1 to hadoop2.
[[email protected] /]$ scp -r /opt/module [email protected]:/opt/module
On hadoop3, copy the software in the /opt/module directory on the hadoop1 server to hadoop3.
[[email protected] opt]$scp -r [email protected]:/opt/module·[email protected]:/opt/module
< /div>
Operate on hadoop3 to copy the software in the /opt/module directory of hadoop1 to hadoop4.
[[emailprotected] opt]$ scp -r [emailprotected]:/opt/module [emailprotected]:/opt/module
Note: the copied /opt/module directory, don’t forget to modify all files on hadoop102, hadoop103, and hadoop104
Owners and owner groups. sudo chown root:root -R /opt/module
Copy the /etc/profile file in hadoop1 to /etc/profile in hadoop2.
[[email protected] ~]$ scp /etc/profile [email protected]:/etc/profile
Note: Don’t forget to source /etc/profile for the copied configuration file.
3, rsync remote synchronization tool
rsync is mainly used for backup and mirroring. It has the advantages of being fast, avoiding copying the same content and supporting symbolic links.
The difference between rsync and scp:
Using rsync to copy files is faster than scp, rsync only works for Update the difference file.
scp copies all files in the past
Basic syntax
rsync -rvl pdir/pdir/pdir/fname [emailprotected]@[emailprotected]: pdir/pdir/pdir/fname
Command Option parameter Path/name of the file to be copied Destination user@Host: Destination path/name
Option parameter description
Case practice< br> Synchronize the /opt/software directory on the hadoop1 machine to the /opt/ directory under the root user of the hadoop2 server
[[emailprotected] opt]$ rsync- rvl /opt/software/ [email protected]:/opt/software
4, xsync Cluster distribution script
Requirement: cyclically copy files to the same directory of all nodes
Requirement analysis:
(a) Original copy of rsync command:
rsync -rvl / opt/module [email protected]:/opt/
(b) Expected script:
The name of the file to be synchronized by xsync
(c) Description: In / The scripts stored in the usr/local/bin directory can be directly executed by other users from anywhere in the system.
Script implementation
(a) Create a file in the /usr/local/bin directory and xsync in the bin directory. The content of the file is as follows:
p>
[[emailprotected] bin]$ touch xsync
[[emailprotected] bin]$ vi xsync
In the file Write the following code in
#!/bin/bash
#1 Get input parameters If there is no parameter, just exit
pcount=$#
if((pcount==0)); then
echo no args;
exit;
fi
#2 Get the file name
p1=$1 fname=`basename
$p1` echo fname=$fname
#3 Get the absolute path of the parent directory
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$ pdir
#4 Get the current user name
user=`whoami`
#5 loop
for((host=3< /span>; host<5; hos t++)); do
echo ------------------- hadoop$host --- -----------
rsync -rvl $pdir/$fname [emailprotected]$host:$pdir
done
(b) Modify the script xsync to have execution permissions
[[email protected] bin]$ chmod 777< /span> xsync
(c) Invoking script form: xsync file name
[[email protected] bin]$ xsync /usr/local/bin
Note: If you put xsync in the /home/user/bin directory, it still can’t Realize global use
You can move xsync to the /usr/local/bin directory.
[[email Protected] /]$ scp -r /opt/module [email Protected]:/opt/module
[[email protected] opt]$scp -r [email protected]:/opt/module·[email protected]:/opt/module
[[emailprotected] opt]$ scp -r [emailprotected]:/opt/module [emailprotected]:/opt/module
[[emailprotected] ~] $ scp /etc/profile [emailprotected]:/etc/profile
[[emailprotected] opt]$ rsync -rvl /opt/software/ [emailprotected]: /opt/software
[[email Protected] bin]$ touch xsync
[[email Protected] bin]$ vi xsync
< p>
#!/bin/bash
#1 Get the number of input parameters, if there are no parameters, exit directly
pcount=$ #
if((pcount==0)); then
echo no args;
exit;
fi
#2 Get the file name
p1=$< span style="color: #800080;">1 fname=`basename
$p1` echo fname=$fname
#3 Get the absolute path of the parent directory
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir
#4 Get the current user name
user=`whoami`
#5 loop
for((host=3 ; host<5; host++)); do
echo ---- --------------- hadoop$host --------------
rsync -rvl $pdir/$fname [emailprotected]$host: $pdir
done
[[email protected] bin]$ chmod 777 xsync[[email protected] bin]$ xsync /usr/local/bin