Third, SerSYNC + RSYNC implements server file real-time synchronization

By Linux

1. Why use rsync+sersync architecture?

1, sersync is developed based on inotify, similar to inotify-tools tools

2, sersync can record changes in the monitored directory (including Add, delete, modify) the name of a specific file or directory, and then when using rsync to synchronize, only the changed files or directories are synchronized.

2. rsync+inotify-tools and rsync+sersync The difference in architecture?

1, rsync+inotify-tools

a, inotify can only record changes in the monitored directory (addition, deletion, modification) and not The specific file or directory that has changed is recorded;

When synchronizing b, rsync, I don’t know which file or directory has changed, and the entire directory is synchronized every time. When the amount of data is large, the entire directory synchronization is very time-consuming (rsync needs to traverse the entire directory to find the comparison file), so the efficiency is very low

2, rsync+sersync< /p>

a, sersync can record the change (addition, deletion, modification) of a specific file or directory name in the monitored directory;

b, rsync only synchronizes when synchronizing The changed file or directory (the data that changes each time is small relative to the entire synchronization directory data, and rsync is very fast when traversing to find the comparison file), so the efficiency is very high.

Summary:

When the amount of synchronized directory data is not large, it is recommended to use rsync+inotify

When the synchronized directory data When there are a lot of files (a few hundred G or more than 1T), it is recommended to use rsync+sersync

#rsync server configuration

1.Deploy rsyncservice

yum install rsync #Install rsync, if you think the version of yum is too low, you can also install it from source

2.vim /etc/rsyncd.conf # DefaultrsyncThere is no configuration file, create one, in the file#and Chinese characters are only comments, use Please clear all comments

#Rsync server
uid = root
gid = root
use chroot = no # Security related
max connections = 2000 # Concurrent Number of connections
timeout = 600 # Timeout time (seconds)
pid file =/var/run/rsyncd.pid # Specify the pid directory of rsync
lock file =/var/run/rsync.lock # Specify rsync lock file [important]
log file = / var/log/rsyncd.log # Specify the log directory for rsync
ignore errors #Ignore some I/O errors
read only = false #Set permissions for rsync file Display the list of rsync server resources
hosts allow = 10.1.0.0/16 #The client IP address that allows data synchronization can be set multiple, separated by commas in English
hosts deny = 0.0.0.0/ 32 #The client IP address that prohibits data synchronization can be set multiple, separated by commas in English state
auth users = rsync_backup #User names that perform data synchronization can be set multiple, separated by commas in English state
secrets file =/etc/rsync.password #User authentication configuration file, which saves the user name and password
##################### #######################
[www] # Module
comment = www
path = /data/www/
############################################ ##
[bbs]
comment = bbs
path = /data/bbs/
###################### #########################
[blog]
comment = blog
path = /data/blog/
#rsync_config____________end

3, create user authentication file

echo “rsync_backup:123456″>/etc/rsync.password #Configuration file, add the following content

4、Set file permissions

chmod 600 /etc/rsync.password

5.Start the daemon and write it to start automatically after booting

6.Create related directories to be synchronized

mkdir -p /data/{www,bbs,blog}

#rsync client configuration

1.An Installrsync, the method is the same as above

2.Create rsyncThe configuration file can be created by the client without content

touch /etc/rsyncd.conf

3.Configure rsyncClient related authorization authentication:

echo “123456 “>/etc/rsync.password
chmod 600 /etc/rsync.password

4.Create the data to be synchronized, in the customer Create some data on the end

mkdir -p /data/{www,bbs,blog}
touch /data/www/www.log /data/bbs/ bbs.log /data/blog/blog.log

5.TestrsyncSynchronous

rsync –daemon
vim /etc/rc.local
# rsync server progress
/usr/bin/rsync –daemon

rsync -avzP /data/www/ [emailprotected]::www –password-file=/etc/rsync.password
#rsync -avzP /backup/ rsync://[emailprotected]/backup/- -password-file=/etc/r sync.password two methods
#rsync -avzP /backup/ rsync://[emailprotected]/backup/test/ –password-file=/etc/rsync.password #test is the directory on the server
Parameter: –delete no difference synchronization
–bwlimit=KB/S speed limit
–exclude=PATTERN exclude files matching PATTERN
–exclude-from=FILE from read exclude patterns >            –include=PATTERN       don’t exclude files matching PATTERN
           –include-from=FILE     read include patterns from FILE

#This step must be successful before proceeding to the next step

6.< strong>Start to deploy sersyncservice

tar fxz sersync2.5.4_64bit_binary_stable_final.tar.gz -C /usr/local/
cd /usr/local/
mv GNU-Linux-x86 sersync

7.Configure sersync

< p>

cp sersync/confxml.xml sersync/confxml.xml-bak
vim sersync/confxml.x ml

Modify the 24–28 line
24 ##控的内容
25 ##Backup server ip and module
26
27
28
The revised content is:
24
25
26
Modified 29-35 line, certification section (RSYNC password authentication)
29
30 ##rsync command parameters
31 # #Authenticated user and password file location
32
33
34
35
The modified content is as follows:
27
28
29 < auth start="true" users="rsync_backup" passwordfile="/etc/rsync.password"/>
30
32
33

8.EnablesersyncData synchronization daemon

< p>

/usr/local/sersync/bin/sersync -d -r -o /usr/local/sersync/conf/confxml.xml
Configure sersync environment variables< br> echo”PATH=$PATH:/usr/local/sersync/”>>/etc/profile
source /etc/profil

After starting the command, the return result is as follows: Normal: **

set the system param
execute: echo 50000000> /proc/sys/fs /inotify/max_user_watches
execute: echo 327679> /proc/sys/fs/inotify/max_queued_events
parse the command param
option: -d run as a daemon
option: -r rsync all the local files to the remote servers before the sersync work
option: -o config xml name: ./confxml.xml
daemon thread num: 10
parse xml config file
XML Parsing error inside file ‘ ./confxml.xml’.
Error: File not found
At line 0, column 0.

Sync test

[[emailprotected] GNU-Linux-x86]# ./sersync2 -d -r -o ./confxml-www.xml

set the system param
execute: echo 50000000> /proc/sys/fs/inotify/max_user_watches
execute: echo 327679> /proc/sys/fs/inotify/max_queued_events
parse the command param
option: -d run as a daemon
option: -r rsync all the local files to the remote servers before the sersync work
option: -o config xml name: ./confxml-www.xml
daemon thr ead num: 10
parse xml config file
host ip: localhost host port: 8008
daemon start, sersync run behind the console
use rsync password-file :
user is rsync_backup
passwordfile is /etc/rsync.password
config xml parse success
please set /etc/rsyncd.conf max connections=0 Manually
sersync working thread 12 = 1(primary thread) + 1(fail retry thread) + 10(daemon sub threads)
Max threads numbers is: 22 = 12(Thread pool nums) + 10(Sub threads)
please according to your cpu, use -n param to adjust the cpu rate
——————————————
rsync the directory recursivly to the remote servers once
working please wait…
execute command: cd /data/www && rsync -artuz -R –delete ./ –timeout=100 [email protected]::www – password-file=/etc/rsync.password >/dev/null 2>&1
run the sersync:
watch path is: /data/www


p>

9.Multi-instance situation

1. Configure multiple confxml.xml files (for example: www, bbs, blog… etc.)
confxml-bbs. xml confxml-blog.xml confxml-www.xml (configurable according to a single instance)
2. Synchronize the corresponding instance files according to different needs
rsync -avzP /data/www/ [emailprotected]:: www/ –password-file=/etc/rsync.password
rsync -avzP /data/bbs/ [email protected]::bbs/ –password-file=/etc/rsync.password
rsync- avzP /data/test/ [email protected]::blog/ –password-file=/etc/rsync.passwor
Just start them separately

< p>Disadvantages of rsync:

1. When a large number of small files are synchronized, the comparison time is longer, and sometimes the rsync process will stop.

2. Synchronize large files, sometimes large files like 10G There will also be problems and will be interrupted. Before the complete synchronization is a hidden file, it can be resumed by parameter

sersync parameter information

parameter-d: enable daemon Mode parameter -r: before monitoring, push the monitored directory and remote host with the rsync command. c parameter -n: specify the number of open daemon threads, the default is 10 parameters -o: specify the configuration file, use the confxml.xml file by default Parameter -m: enable other modules separately, use -m refreshCDN to open the refresh CDN module parameter -m: enable other modules separately, use -m socket to open the socket module parameter -m: enable other modules separately, use -m http to open the http module Add the -m parameter, the synchronization program will be executed by default

, , , , , , , ,

Leave a Comment

Your email address will not be published.