HeartBeat + DRBD + MySQL high available architecture deployment - architecture, available, deployment, drbd, Heartbeat, Heartbeat + DRBD + mysql, high, mysql

This article was earned from http://shanhu.blog.51cto.com/1293405/1212605

Heartbeat introduction

Official site: http://linux-ha.org/wiki/Main_Page

< p style="padding:0px;margin-top:0px;margin-bottom:0px;clear:both;height:auto;color:rgb(44,44,44);font-family:'宋体','Arial Narrow ', arial, serif;font-size:14px;line-height:28px;white-space:normal;background-color:rgb(255,255,255);"> Heartbeat can quickly transfer resources (VIP addresses and program services) from one faulty server to another normal server to provide services, heartbeat is similar to keepalived, heartbeat can implement the failover function, but it cannot implement the health check on the backend

DRBD introduction

Official site: http://www.drbd.org/ p>

DRBD (Distributed Replicated Block Device) is a software that directly synchronizes and mirrors data at the remote server based on the block device level. It is implemented by software, without sharing, a storage replication solution for the contents of mirrored block devices between servers. It can realize real-time mirroring or synchronous replication based on block device level between two servers in the network (both servers are successfully written)/asynchronous replication (local server is successfully written), which is equivalent to RAID1 of the network , Because it is based on block devices (disks, LVM logical volumes), At the bottom of the file system, so data copying is faster than the cp command

DRBD has been approved by MySQL The official document manual is written as one of the recommended high-availability solutions

Introduction to MySQL

Official site: http://www.mysql.com/

MySQL is an open source small relational database managementsystem . Currently MySQL is widely usedapplicationIn small and medium-sized websites on the Internet. Because of itssizeSmall, fast, low total cost of ownership, especially the open source feature, many small and medium-sized websites in order to reduce the overall website ownership Cost and chose MySQL as the websitedatabase.

Heartbeat and keepalived application scenarios and differences

Many netizens say why they don’t use keepalived and use heartbeat, which is not updated for a long time. Let’s talk about the application scenarios and differences between them.

1. For web, db, load balancing (lvs, haproxy, nginx), etc., both heartbeat and keepalived can be implemented< /p>

2, lvs is best combined with keepalived, because keepalived was originally generated for lvs, (heartbeat does not have a health check function for RS, heartbeat can pass ldircetord To perform the health check function)

3, mysql dual master and multiple slaves, NFS/ MFS storage is characterized by the need for data synchronization. This kind of business is best to use heartbeat, because heartbeat has its own drbd script

Summary: High-availability applications without data synchronization can choose keepalived, and high-availability applications with data synchronization can choose heartbea

1. Environment

System: CentOS 6.4×64 minimal installation

ha-node1: 192.168 .3.71

ha-nod e2: 192.168.3.72

vip: 192.168.3.73

mysql: 192.168.3.74

Requirements:

1. After the main library ha-node1 goes down, ha-node2 automatically takes over VIP and all slaves Library

2. When ha-node2 takes over, it will not affect the master-slave synchronization replication of the slave library

Disk capacity Mount point Description

/dev/sdb1 8G /data Store data

/dev/sdb2 2G Store DRBD synchronization status information

1. The metadata partition must not be formatted to establish a file system (sdb2 stores the status information of drbd synchronization)

2. Do not mount the divided partitions

3, the production environment DRBD meta data partition can generally be set to 1 -2G, the data partition depends on the maximum demand

4. Two hard drives are the same size in a production environment< /p>

Second, basic configuration

node1 and node2 The operation is the same

#Close iptables and selinux[root@ha-node1 ~]# service iptables stop[root@ha-node1~] #GetenforceDisabled #Make sure the result is correct [root@ha-node1~]#wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm[root@ha -node1 ~]# rpm -ivh epel-release-6-8.noarch.rpmwarning: epel-release-6-8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEYPreparing...### ####################################[100%] 1:epel-release# #########################################[100%][root@ha -node1~]#rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6[root@ha-node1~]#rpm -i vh http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpmRetrieving http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpmwarning:/var /tmp/rpm-tmp.zglfTu: Header V4 DSA/SHA1 Signature, key ID baadae52: NOKEYPreparing...######################## ################ [100%] 1: elrepo-release ###################### ##################[100%]#Configure local host resolution [root@ha-node1~]# echo "192.168.3.71 ha-node1" >>/etc /hosts[root@ha-node1~]#echo"192.168.3.72ha-node2">>/etc/hosts[root@ha-node1~]#cat/etc/hosts127.0.0.1 localhostlocalhost.localdomainlocalhost4localhost4 .localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.3.71 ha-node1192.168.3.72 ha-node2# Configure ssh mutual trust, here only the configuration process of ha-node1 [root@ha-node1] #Ssh-keygen[root@ha-node1~]# ssh-copy-id -i~/.ssh/id_rsa.pub root@ha-node2

Three, install and configure heartbeat

(1). Install heartbeat

# Perform installation operations on both ha-node1 and ha-node2 [root @ha-node1 ~]# yum install heartbeat -y

(2).Configurationha.cf

[root@ha-node1 ~]# cd /usr/share/doc/heartbeat-3.0.4/[root @ha-node1 heartbeat-3.0.4]# cp authkeys ha.cf haresources /etc/ha.d/[root@ha-node1~]# grep-v"^#"/etc/ha.d/ha.cf logfile / var / log / ha-loglogfacility local1keepalive 2deadtime 30warntime 10initdead 120mcast eth0 225.0.10.1 694 1 0 # multicast ha node2-# ha-node1 # standby node the master node node auto_failback onnode crm no

(3). Configure authkeys

[ root@ha-node1 ~]# dd if=/dev/random bs=512 count=1| openssl md50+1 records in0+1 records out19 bytes (19B) copied, 4.6784e-05 s, 406 kB/s( stdin)=317fa30c3a96a0e54b8018d3e0f3c04a[root@ha- node1~]# grep -v ^# /etc/ha.d/authkeys auth 11 md5 317fa30c3a96a0e54b8018d3e0f3c04a #Use md5 certification#Modify the authentication file permissions to 600[root@ha-etc1mod/#] /authkeys

(4). Configure haresource

[root@ha-node1~]# grep-v^ #/Etc/ha.d/haresources ha-node1 IPaddr::192.168.3.73/24/eth0 #here only one VIP resource is configured here temporarily

(5). Start heartbeat

[root@ha-node1~]#/etc/ init.d/heartbeat startStarting High-Availability services: INFO: Resource is stoppedDone.[root@ha-node1 ~]# chkconfig heartbeat off Description: Turn off the automatic startup when the server restarts, you need to manually start #ha-node2 The configuration file is copied from ha-node1 and the service is started [root@ha-node1 ha.d]#scp authkeys ha.cf haresources ha-node2:/etc/ha.d/#View the results [root@ha-node1 ~]#ip a|grep 192.168.3.73 #vip on the master node inet 192.168.3.73/24 brd 192.168.3.2 55 scope global secondary eth0[root@ha-node2~]#ip a|grep 192.168.3.73 #There is no vip on the standby node

(6). Test heartbeat

Normal state

#ha- IP information of node1[root@ha-node1~]#ip a|grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet global 192.168.3.71/24 brd 192.168.3.255 scope eth0 inet 192.168.3.73/24 brd 192.168.3.255 scope global secondary eth0#ha-node2's IP information[root@ha-node2~]# ip a|grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.3.72/24 brd 192.168.3.255 scope global eth0

simulation Status information after the master node is down

#Stop the heartbeat service on the master node ha0node1 [root@ha-node1 ~]# /etc/init.d/heartbeat stopStopping High-Availability services: Done.[root@ha-node1~]# ip a|grep eth0 #The heartbea of the master node After the service was stopped, VIP resources were robbed 2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.3.71/24 brd 192.168.3.255 scope global eth0 #On the standby node ha- node2 View resources[root@ha-node2~]#ip a|grep eth0 #The standby node has automatically taken over the resource 2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000 192.168.3.7 24 brd 192.168.3.255 scope global eth0 inet 192.168.3.73/24 brd 192.168.3.255 scope global secondary eth0

Restore the heartbeat service of the master node

[root@ha-node1 ~]# /etc/init.d/heartbeat startStarting High- Availability services: INFO: Resource is stoppedDone.# After the heartbeat service of the master node is restored, the resource is taken back [root@ha-node1]# ip a |grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 192.168.3.71/24 brd 192.168.3.255 scope global eth0 inet 192.168.3.73/24 brd 192.168.3.255 scope global secondary eth0 #View standby node[root@ha -node2~]#ip a|grep eth0 #vip resource has been removed 2: eth0:  global mtu 1500 qdisc pfifo_fast state UP qlen 1000 192.168.3.25 inet 192.168.3.72/24 brd

Four, install and deploy DRBD

< span style="font-size:14px;font-family:'宋体', SimSun;">(1). Partition the hard disk, the operation of ha-node1 and ha-node2 is the same

[root@ha-node1 ~]# fdisk/dev/sdb# Description: /dev/sdb is divided into 2 partitions /dev/sdb1 and /dev/sdb2 ，/Dev/sdb1=8G[root@ha-node1~]# partprobe /dev/sdb#Format the partition [root@ha-node1~]# mkfs.ext4/dev/sdb1 Description: sdb2 partition is meta data Partition, no formatting operation required [root@ha-node1 ~]# tune2fs -c -1 /dev/sdb1 Description: Set the maximum number of mounts to -1, turn off the mandatory check and mount limit

(2). Install DRBD

Since our system is CentOS6.4, we also need to install the kernel module. The version needs to be consistent with uname -r. The installation package is extracted from the system installation software. , The process is omitted. The installation process of ha-node1 and ha-node2 is the same, here only the installation process of ha-node1 is given

#Install the kernel file [root @ha-node1 ~]# rpm -ivh kernel-devel-2.6.32-358.el6.x86_64.rpm kernel-headers-2.6.32-358.el6.x86_64.rpm [root@ha-node1 ~]#yum install drbd84 kmod-drbd84 -y

(3). Configure DRBD

a. Modify the global configuration file

[root@ha-node1 ~]#egrep-v"^$ |^#|^[[:space:]]+#"/etc/drbd.d/global_common.conf global {usage-count no;}common {protocol C; handlers {} startup{} options{} disk{ on -io-error detach; no-disk-flushes; no-md-flushes; rate 200M;} net {sndbuf-size 512k; max-buffers 8000; unplug-watermark 1024; max-epoch-size 8000; cram-hmac- alg "sha1"; shared-secret "weyee2014"; after-sb-0pridisconnect; after-sb-1pridisconnect; after-sb-2pridis connect; rr-conflict disconnect; })

b. Increase resources

[root@ha-node1 ~]# cat /etc/drbd.d/mysqldata.resresource mysqldata {on ha-node1 {device/dev/drbd1; disk / dev/sdb1; address 192.168.3.71:7789; meta-disk /dev/sdb2 [0];} on ha-node2 {device/dev/drbd1; disk /dev/sdb1; address 192.168.3.72:7789; meta-disk /dev/sdb2 [0]; }}

c. Copy the configuration file to ha-node2 , Restart the system to load the drbd module, and initialize the meta data

[root@ha-node1 drbd.d]#scp global_common.conf mysqldata. res ha-node2:/etc/drbd.d/[root@ha-node1~]#depmod[root@ha-node1~]#modprobedrbd[root@ha-node1~]#lsmod|grep drbddrbd 246 rc 931c 931c 1 drbd#Initialize meta data in ha-node1[root@ha-node1~]# drbdadm create-md mysqldatainitializing activity logNOT initializing bitmapWriting metadata...New drbd meta d ata block successfully created.#Load the module on ha-node2 and initialize the meta data[root@ha-node2~]#depmod[root@ha-node2~]#modprobedrbd[root@ha-node2~]#lsmod|grep drbddrbd 365931 0 libcrc32c 1246 1 drbd[root@ha-node2~]# drbdadm create-md mysqldatainitial activity blockdatadatablockafully font-size:14px;font-family:'宋体', SimSun;">d. Start drbd on ha-node1 and ha-node2
 #ha-node1操作[[email protected]]#/etc/init.d/drbd startStarting DRBD resources: [create res: mysqldata] prepare disk: mysqldata# adjust disk: mysqldata# mysqldataadjust net: mysqldata# ha-node2 operation[root@ha-node2~]# /etc/init.d/drbd startStarting DRBD resources: [}}}}}}}} Res: mysqldata â€“ prepare disk: mysqldata â€“ adjust disk: mysqldata â€“ adjust net: mysqldata 1 ]# drbd-overview 1:mysqldata/0 Connected Secondary/Secondary Inconsistent/Inconsistent # will ha- node1 is set as the primary node[root@ha-node1~]# drbdadm---overwrite-data-of-peer primary mysqldata[root@ha-node1~]#drbd-overview 1:mysqldata/0 SyncSource Primary/Secondary UpToDate /Inconsistent [>....................] sync'ed: 4.3% (7848/8196)M #Mount the DRBD device to the /data directory and write Enter test data ha-node2.txt[root@ha-node1~]#mount/dev/drbddrbd/drbd1[root@ha-node1~]#mount/dev/drbd1/data/[root@ha-node1~]# touch /data/ha-node2.txt[root@ha-node1~]#ll /data/total 16-rw-r--r-- 1 root root 0 Jun 12 16:35 ha-node2.txtdrwx---- --- 2 root root 16384 Jun 12 15:31 lost+found# The status result shows UpToDate/UpToDate indicating that the data of the primary and standby nodes has been synchronized [root@ha-node1 ~]# drbd-overview 1:mysqldata/0 Connected Primary/Secondary UpToDate/UpToDate /data ext4 7.8G 19M 7.4G 1%
e. Test DRBD
Normal state

[root@ha-node1~]# drbd-overview 1:mysqldata/0 Connected Primary/Secondary UpToDate/UpToDat e /data ext4 7.8G 19M 7.4G 1%[root@ha-node2~]# drbd-overview 1:mysqldata/0 Connected Secondary/Primary UpToDate/UpToDate#Note: here is that ha-node1 is the primary node, ha -node2 is the slave node
simulate the state after the downtime

[root@ha-node1]~ ]#Umount/data#Set the status of ha-node1 to Secondary[root@ha-node1~]# drbdadm secondary mysqldata[root@ha-node1~]# drbd-overview 1:mysqldata/0 ConnectedSecondary/SecondaryUpToDate/ UpToDate #Set the status of ha-node2 to primary[root@ha-node2~]# drbdadm primary mysqldata[root@ha-node2~]#drbd-overview 1:mysqldata/0 Connected Primary/Secondary UpToDate/UpToDate[root@ ha-node2~]#mount/dev/drbddrbd/drbd1 [root@ha-node2~]#mount/dev/drbd1/mnt#View the file, the test result is normal [root@ha-node2~]#ll/mnttotal 16- rw-r--r-- 1 root root 0 Jun 12 16:35 ha-node1.txt-rw-r--r-- 1 root root 0 Jun 15 09:54 ha-node2.txt#Note: DRBD main After the node is down, the standby node can be used normally after it is set to the primary state, and the data is consistent# DRBD state Restore to the original state
Five. Install and deploy mysql and mysql on ha-node1 and ha-node2 Version 5.5.37
ha-node1:
< /p> 
#Create users and groups[root@ha-node1~]# groupadd mysql[root@ha-node1~]# useradd-g mysql mysql-s/ sbin/nologin#Create a data directory[root@ha-node1~]#mkdir-p/data[root@ha-node1~]#mount/dev/drbd1/data/#Mount the DRBD device to the /data directory[ root@ha-node1~]#mkdir-p/data/mysql/data#Unzip and install mysql[root@ha-node1~]#yum-y install make gcc-c++cmakebison-devel ncurses-devel[root@ha -node1~]#tar xf mysql-5.5.37.tar.gz[root@ha-node1~]# cd mysql-5.5.37[root@ha-node1 mysql-5.5.37]#cmake\>-DCMAKE_INSTALL_PREFIX= /usr/local/mysql-5.5.37\>-DMYSQL_DATADIR=/data/mysql/data\>-DSYSCONFDIR=/etc\>-DWITH_MYISAM_STORAGE_ENGINE=1\>-DWITH_INNOBASE_STORAGE_ENGINE=1LINEINE=1-DWITH_MEMORYD_READ=1 =1\>-DMYSQL_UNIX_ADDR=/var/lib/mysql/mysql.sock\>-DMYSQL_TCP_ PORT=3306\>-DENABLED_LOCAL_INFILE=1\>-DWITH_PARTITION_STORAGE_ENGINE=1\>-DEXTRA_CHARSETS=all\>-DDEFAULT_CHARSET=utf8\>-DDEFAULT_COLLATION=utf8_general_cimake [email protected] #Initial data[root@ha-node1 mysql-5.5.37]# cd /usr/local/mysql-5.5.37/[root@ha-node1 mysql-5.5.37]# scripts/mysql_install_db --datadir=/data /mysql/data/--user=mysql--basedir=/usr/local/mysql-5.5.37/Installing MySQL system tables...OKFilling help tables...OKTo start mysqld at boot time you have to copysupport-files /mysql.server to the right place for your systemPLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER! To do so, start the server, then issue the following commands: /usr/local/mysql-5.5.37//bin/ mysqladmin -u root password'new-password'/usr/local/mysql-5.5.37//bin/mysqladmin -u root -h ha-node1 password'new-password'Alternatively you can run:/usr/local/mysql -5.5.37//bin/mysql_secure_installationwhich will also give you the option of removing the testdatabases and anon ymous user created by default.  This isstrongly recommended for production servers.See the manual for more instructions.You can start the MySQL daemon with:cd /usr/local/mysql-5.5.37/ ; /usr/local/mysql-5.5.37//bin/mysqld_safe &You can test the MySQL daemon with mysql-test-run.plcd /usr/local/mysql-5.5.37//mysql-test ; perl mysql-test-run.plPlease report any problems at #复制mysql配置文件[root@ha-node1 mysql-5.5.37]# cp -rf support-files/my-large.cnf /etc/my.cnf  #创建启动脚本[root@ha-node1 mysql-5.5.37]# cp support-files/mysql.server /etc/init.d/mysqld[root@ha-node1 mysql-5.5.37]# chmod +x /etc/init.d/mysqld#配置软连接[root@ha-node1 mysql-5.5.37]# ln -s /usr/local/mysql-5.5.37/ /usr/local/mysql[root@ha-node1 mysql-5.5.37]# ln -s /usr/local/mysql-5.5.37/bin/* /usr/sbin/#启动mysql[root@ha-node1 ~]# /etc/init.d/mysqld startStarting MySQL.. SUCCESS! [root@ha-node1 ~]# netstat -anpt |grep mysqltcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      17309/mysqld #连接到mysql并创建数据库dev[root@ha-node1 ~]# mysqlWelcome to the MySQL monitor.  Commands end with ; or \g.Your MySQL connection id is 1Server version: 5.5.37-log Source distributionCopyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> create database dev;Query OK, 1 row affected (0.00 sec)mysql> show databases;+--------------------+| Database           |+--------------------+| information_schema || dev                |            #这是刚创建的数据库| mysql              || performance_schema || test               |+--------------------+5 rows in set (0.00 sec)#关闭mysql，设置开机不启动[root@ha-node1 ~]# /etc/init.d/mysqld stopShutting down MySQL. SUCCESS! [root@ha-node1 ~]# chkconfig --add mysqld[root@ha-node1 ~]# chkconfig mysqld off
ha-node2:
#由于我们使用的是DRBD方式，所以这里理论上只需要安装好mysql就能正常启动了，且能识别到我们在ha-node1上创建的数据库dev#确认ha-node2的DRBD状态[root@ha-node2 ~]# drbd-overview  1:mysqldata/0  Connected Secondary/Primary UpToDate/UpToDate #停止DRBD[root@ha-node2 ~]# drbdadm down mysqldata#安装基础软件包[root@ha-node2 ~]# yum -y install make gcc-c++ cmake bison-devel  ncurses-devel#创建用户和组[root@ha-node2 ~]# groupadd mysql[root@ha-node2 ~]# useradd -g mysql mysql -s /sbin/nologin#创建数据目录[root@ha-node2 ~]# mkdir -p /data[root@ha-node2 ~]# mount /dev/sdb1 /data        #这里的/dev/sdb1是DRBD的设备#解压并安装mysql[root@ha-node2 ~]# tar xf mysql-5.5.37.tar.gz [root@ha-node2 ~]# cd mysql-5.5.37[root@ha-node2 mysql-5.5.37]# cmake \> -DCMAKE_INSTALL_PREFIX=/usr/local/mysql-5.5.37 \> -DMYSQL_DATADIR=/data/mysql/data \> -DSYSCONFDIR=/etc \> -DWITH_MYISAM_STORAGE_ENGINE=1 \> -DWITH_INNOBASE_STORAGE_ENGINE=1 \> -DWITH_MEMORY_STORAGE_ENGINE=1 \> -DWITH_READLINE=1 \> -DMYSQL_UNIX_ADDR=/var/lib/mysql/mysql.sock \> -D MYSQL_TCP_PORT=3306 \> -DENABLED_LOCAL_INFILE=1 \> -DWITH_PARTITION_STORAGE_ENGINE=1 \> -DEXTRA_CHARSETS=all \> -DDEFAULT_CHARSET=utf8 \> -DDEFAULT_COLLATION=utf8_general_ci[root@ha-node2 mysql-5.5.37]# make && make install#mysql的配置文件和启动脚本从ha-node1上复制过来[root@ha-node1 ~]# scp /etc/my.cnf ha-node2:/etc   [root@ha-node1 ~]# scp /etc/init.d/mysqld ha-node2:/etc/init.d/#配置软连接[root@ha-node2 ~]# ln -s /usr/local/mysql-5.5.37/ /usr/local/mysql[root@ha-node2 ~]# ln -s /usr/local/mysql-5.5.37/bin/* /usr/sbin/#设置mysql开机不启动[root@ha-node2 ~]# chkconfig --add mysqld[root@ha-node2 ~]# chkconfig mysqld off#启动mysql[root@ha-node2 ~]# /etc/init.d/mysqld startStarting MySQL.. SUCCESS! [root@ha-node2 ~]# netstat -anpt |grep mysqltcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      16617/mysqld #连接到mysql，查看是否有在ha-node1上创建的dev数据库[root@ha-node2 ~]# mysqlWelcome to the MySQL monitor.  Commands end with ; or \g.Your MySQL connection id is 1Server version: 5.5.37-log Source dis tributionCopyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> show databases;+--------------------+| Database           |+--------------------+| information_schema || dev                |        #在ha-node1上创建的数据库能正常显示出来| mysql              || performance_schema || test               |+--------------------+5 rows in set (0.05 sec)#关闭mysql,卸载/data分区，启动DRBD设备[root@ha-node2 ~]# /etc/init.d/mysqld stopShutting down MySQL. SUCCESS! [root@ha-node2 ~]# umount /mntumount: /mnt: not mounted[root@ha-node2 ~]# umount /data/[root@ha-node2 ~]# drbdadm up mysqldata[root@ha-node2 ~]# drbd-overview  1:mysqldata/0  Connected Secondary/Primary UpToDate/UpToDate
到此基于DRBD的mysql安装完成
六、配置heartbeat调用DRBD
确保ha-node1和ha-node2没有挂载DRBD设备，且ha-node1是主节点
#ha-node1的挂载情况[root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot#ha-node2的挂载情况[root@ha-node2 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.7G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot[root@ha-node1 ~]# drbd-overview  1:mysqldata/0  Connected Primary/Secondary UpToDate/UpToDate
前面我们通过heartbeat管理VIP已经成功，这里我们配置管理DRBD设备

编辑配置文件/etc/ha.d/haresources
[root@ha-node1 ~]# grep -v ^# /etc/ha.d/haresources ha-node1 IPaddr::192.168.3.73/24/eth0 drbddisk::mysqldata Filesystem::/dev/drbd1::/data::ext4#说明drbddisk::mysqldata      <==启动drbd mysqldata资源，相当于执行/etc/ha.d/resource.d/drbddisk mysqldata stop/start操作Filesystem::/dev/drbd1::/data::ext4     <==drbd分区挂载到/data目录，相当于执行/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext4 stop/start        <==相当于系统中执行mount /dev/drbd1 /data#将资源文件复制到ha-node2上[root@ha-node1 ~]# scp /etc/ha.d/haresources ha-node2:/etc/ha.d/#ha-node1重新加载DRBD资源前的挂载情况[root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot#在ha-node1上停止heartbeat资源并重新启动[root@ha-node1 ~]# /etc/init.d/stop[root@ha-node2 ~]# /etc/init.d/stop[root@ha-node1 ~]# /etc/init.d/ heartbeat startStarting High-Availability services: INFO:  Resource is stoppedDone.[root@ha-node2 ~]# /etc/init.d/heartbeat startStarting High-Availability services: INFO:  Resource is stoppedDone.#查看结果[root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot/dev/drbd1            7.8G   48M  7.4G   1% /data        #DRBD设备在主节点正常挂载到/data目录下#将ha-node1的heartbeat服务停止[root@ha-node1 ~]# /etc/init.d/heartbeat stopStopping High-Availability services: Done.[root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot#在ha-node2上查看挂载情况[root@ha-node2 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.7G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot/dev/drbd1            7.8G   48M  7.4G   1% /data        #结果正常#查看备节点接管日志[root@ha-node2 ~]# tail -20 /var/log/messagesJun 15 11:30:10 ha-node2 ResourceManager(default)[17855]: info: Acquiring resource group: ha-node1 IPaddr::192.168.3.73/24/eth0 drbddisk::mysqldata Filesystem::/dev/drbd1::/data::ext4Jun 15 11:30:10 ha-node2 /usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.3.73)[17883]: INFO:  Resource is stoppedJun 15 11:30:10 ha-node2 ResourceManager(default)[17855]: info: Running /etc/ha.d/resource.d/IPaddr 192.168.3.73/24/eth0 startJun 15 11:30:10 ha-node2 IPaddr(IPaddr_192.168.3.73)[18006]: INFO: Adding inet address 192.168.3.73/24 with broadcast address 192.168.3.255 to device eth0Jun 15 11:30:10 ha-node2 IPaddr(IPaddr_192.168.3.73)[18006]: INFO: Bringing device eth0 upJun 15 11:30:11 ha-node2 IPaddr(IPaddr_192.168.3.73)[18006]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/re source-agents/send_arp-192.168.3.73 eth0 192.168.3.73 auto not_used not_usedJun 15 11:30:11 ha-node2 /usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.3.73)[17980]: INFO:  SuccessJun 15 11:30:11 ha-node2 ResourceManager(default)[17855]: info: Running /etc/ha.d/resource.d/drbddisk mysqldata startJun 15 11:30:11 ha-node2 kernel: block drbd1: role( Secondary -> Primary ) Jun 15 11:30:11 ha-node2 /usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[18138]: INFO:  Resource is stoppedJun 15 11:30:11 ha-node2 ResourceManager(default)[17855]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext4 startJun 15 11:30:11 ha-node2 Filesystem(Filesystem_/dev/drbd1)[18222]: INFO: Running start for /dev/drbd1 on /dataJun 15 11:30:11 ha-node2 kernel: EXT4-fs (drbd1): mounted filesystem with ordered data mode. Opts: Jun 15 11:30:11 ha-node2 /usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[18214]: INFO:  SuccessJun 15 11:30:11 ha-node2 mach_down(def ault)[17828]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquiredJun 15 11:30:11 ha-node2 mach_down(default)[17828]: info: mach_down takeover complete for node ha-node1.Jun 15 11:30:11 ha-node2 heartbeat: [17726]: info: mach_down takeover complete.Jun 15 11:30:41 ha-node2 heartbeat: [17726]: WARN: node ha-node1: is deadJun 15 11:30:41 ha-node2 heartbeat: [17726]: info: Dead node ha-node1 gave up resources.Jun 15 11:30:41 ha-node2 heartbeat: [17726]: info: Link ha-node1:eth0 dead.#我们在重新启动ha-node1的heartbeat服务[root@ha-node1 ~]# /etc/init.d/heartbeat startStarting High-Availability services: INFO:  Resource is stoppedDone.#ha-node1的挂载情况[root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot/dev/drbd1            7.8G   48M  7.4G   1% /data#ha-node2的挂载情况[root@ha-node2 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.7G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot
最后配置heartbeat+DRBD+mysql的组合
编辑配置文件/etc/ha.d/haresources
[root@ha-node1 ~]# grep -v ^# /etc/ha.d/haresources ha-node1 IPaddr::192.168.3.73/24/eth0 drbddisk::mysqldata Filesystem::/dev/drbd1::/data::ext4 mysqld#说明drbddisk::mysqldata      <==启动drbd mysqldata资源，相当于执行/etc/ha.d/resource.d/drbddisk mysqldata stop/start操作Filesystem::/dev/drbd1::/data::ext4     <==drbd分区挂载到/data目录，相当于执行/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext4 stop/start        <==相当于系统中执行mount /dev/drbd1 /datamysqld               <==启动mysql服务脚本，相当于/etc/init.d/mysqld stop/start#把配置文件复制到ha-node2上[root@ha-node1 ~]# scp /etc/ha.d/haresources ha-node2:/etc/ha.d/
 重启ha-node1的heartbeat服务
 [root@ha-node1 ~]# /etc/init.d/heartbeat stop[root@ha-node1 ~]# /etc/init.d/heartbeat startStarting High-Availability services: INFO:  Resource is stoppedDone.#查看结果，正常情况下vip,DRBD设备，mysql服务，都应该在ha-node1上正常启动[root@ha-node1 ~]# netstat -anpt |grep mysqltcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      21075/mysqld        [root@ha-node1 ~]# ip a|grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000    inet 192.168.3.71/24 brd 192.168.3.255 scope global eth0    inet 192.168.3.73/24 brd 192.168.3.255 scope global secondary eth0[root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot/dev/drbd1            7.8G   48M  7.4G   1% /data#测试mysql是否正常[root@ha-node1 ~]# mysqlWelcome to the MySQL monitor.  Commands end with ; or \g.Your MySQL connection id is 1Server version: 5.5.37-log Source distributionCopyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or itsaffiliates. Other names may be trademarks of their respectiveowners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> create database yanfa;Query OK, 1 row affected (0.02 sec)mysql> show databases;+--------------------+| Database           |+--------------------+| information_schema || dev                || mysql              || performance_schema || test               || yanfa              |        #创建一个新的数据库研发+--------------------+6 rows in set (0.02 sec)
重启下ha-node2的heartbeat服务
#我们修改配置文件后，还没有重启ha-node2的heartbeat服务，这里我们重启下[root@ha-node2 ~]# /etc/init.d/heartbeat stopStopping High-Availability services: Done.[root@ha-node2 ~]# /etc/init.d/heartbeat startStarting High-Availability services: INFO:  Resource is stoppedDone.
测试mysql，是否能正常漂移并提供服务
#现在我们的vip,DRBD,mysql服务都是运行在ha-node1上的，我们停止ha-node1的heartbeat服务后，这些资源应该会被ha-node2自动接管[root@ha-node1 ~]# /etc/init.d/heartbeat stopStopping High-Availability services: Done.[root@ha-node1 ~]# ip a|grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000    inet 192.168.3.71/24 brd 192.168.3.255 scope global eth0[root@ha-node1 ~]# netstat -anpt |grep mysql[root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot#从上面的结果能看出所有资源已经被移除#在node2上查看结果[root@ha-node2 ~]# ip a|grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000    inet 192.168.3.72/24 brd 192.168.3.255 scope global eth0    inet 192.168.3.73/24 brd 192.168.3.255 scope global secondary eth0[root@ha-node2 ~]# netstat -anpt |grep mysqltcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      21149/mysqld        [root@ha-node2 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.7G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot/dev/drbd1            7.8G   48M  7.4G   1% /data#结果显示ha-node2已正常接管所有资源#查看mysql结果是否正常[root@ha-node2 ~]# mysql -e "show databases;"+--------------------+| Database           |+--------------------+| information_schema || dev                || mysql              | | performance_schema || test               || yanfa              |        #刚才创建的yanfa数据库能够正常显示+--------------------+#查看ha-node2的日志文件[root@ha-node2 ~]# tail -30 /var/log/messagesJun 15 11:46:17 ha-node2 kernel: block drbd1: peer( Primary -> Secondary ) Jun 15 11:46:17 ha-node2 heartbeat: [20281]: info: Received shutdown notice from 'ha-node1'.Jun 15 11:46:17 ha-node2 heartbeat: [20281]: info: Resources being acquired from ha-node1.Jun 15 11:46:17 ha-node2 heartbeat: [20321]: info: acquire local HA resources (standby).Jun 15 11:46:17 ha-node2 heartbeat: [20322]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys ha-node2] to acquire.Jun 15 11:46:17 ha-node2 heartbeat: [20321]: info: local HA resource acquisition completed (standby).Jun 15 11:46:17 ha-node2 heartbeat: [20281]: info: Standby resource acquisition done [foreign].Jun 15 11:46:17 ha-node2 harc(default)[20347]: info: Running /etc/ha.d//rc.d/status statusJun 15 11:46:17 ha-node2 mach_down(default)[20364]: in fo: Taking over resource group IPaddr::192.168.3.73/24/eth0Jun 15 11:46:17 ha-node2 ResourceManager(default)[20391]: info: Acquiring resource group: ha-node1 IPaddr::192.168.3.73/24/eth0 drbddisk::mysqldata Filesystem::/dev/drbd1::/data::ext4 mysqldJun 15 11:46:17 ha-node2 /usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_192.168.3.73)[20419]: INFO:  Resource is stoppedJun 15 11:46:17 ha-node2 ResourceManager(default)[20391]: info: Running /etc/ha.d/resource.d/IPaddr 192.168.3.73/24/eth0 startJun 15 11:46:17 ha-node2 IPaddr(IPaddr_192.168.3.73)[20542]: INFO: Adding inet address 192.168.3.73/24 with broadcast address 192.168.3.255 to device eth0Jun 15 11:46:17 ha-node2 IPaddr(IPaddr_192.168.3.73)[20542]: INFO: Bringing device eth0 upJun 15 11:46:17 ha-node2 IPaddr(IPaddr_192.168.3.73)[20542]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.3.73 eth0 192.168.3.73 auto not_used not_usedJun 15 11:46:17 ha-node2 /usr/lib/ocf/resource.d//heartbea t/IPaddr(IPaddr_192.168.3.73)[20516]: INFO:  SuccessJun 15 11:46:17 ha-node2 ResourceManager(default)[20391]: info: Running /etc/ha.d/resource.d/drbddisk mysqldata startJun 15 11:46:17 ha-node2 kernel: block drbd1: role( Secondary -> Primary ) Jun 15 11:46:17 ha-node2 /usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[20670]: INFO:  Resource is stoppedJun 15 11:46:17 ha-node2 ResourceManager(default)[20391]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd1 /data ext4 startJun 15 11:46:17 ha-node2 Filesystem(Filesystem_/dev/drbd1)[20752]: INFO: Running start for /dev/drbd1 on /dataJun 15 11:46:17 ha-node2 kernel: EXT4-fs (drbd1): mounted filesystem with ordered data mode. Opts: Jun 15 11:46:17 ha-node2 /usr/lib/ocf/resource.d//heartbeat/Filesystem(Filesystem_/dev/drbd1)[20744]: INFO:  SuccessJun 15 11:46:17 ha-node2 ResourceManager(default)[20391]: info: Running /etc/init.d/mysqld  startJun 15 11:46:19 ha-node2 mach_down(default)[20364]: info: /usr/share/heartb eat/mach_down: nice_failback: foreign resources acquiredJun 15 11:46:19 ha-node2 heartbeat: [20281]: info: mach_down takeover complete.Jun 15 11:46:19 ha-node2 mach_down(default)[20364]: info: mach_down takeover complete for node ha-node1.Jun 15 11:46:49 ha-node2 heartbeat: [20281]: WARN: node ha-node1: is deadJun 15 11:46:49 ha-node2 heartbeat: [20281]: info: Dead node ha-node1 gave up resources.Jun 15 11:46:49 ha-node2 heartbeat: [20281]: info: Link ha-node1:eth0 dead.
我们在ha-node1上重新启动heartbeat服务，查看资源是否会被接管回来
[root@ha-node1 ~]# /etc/init.d/heartbeat startStarting High-Availability services: INFO:  Resource is stoppedDone.#查看结果，资源能够被正常接管回来[root@ha-node1 ~]# ip a|grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000    inet 192.168.3.71/24 brd 192.168.3.255 scope global eth0    inet 192.168.3.73/24 brd 192.168.3.255 scope global sec ondary eth0[root@ha-node1 ~]# netstat -anpt |grep mysqltcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      22805/mysqld        [root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot/dev/drbd1            7.8G   48M  7.4G   1% /data
七、配置mysql高可用主从
我们在上面的架构基础上增加一个mysql的从服务器，实现读写分离。从服务器的安装我们使用脚本安装
脚本内同如下
#!/bin/bash DATADIR='/data/mysql/data'VERSION='mysql-5.5.37'export LANG=zh_CN.UTF-8 #Source function library.. /etc/init.d/functions #camke install mysql5.5.Xinstall_mysql(){                #这里的密码需要我们手动输入，设置密码是lyao36843        read -p "please input a password for root: " PASSWD        if [ ! -d $DATADIR ];then                mkdir -p $DATADIR        fi        yum install cmake make gcc-c++ bison-devel ncurses-devel -y        id mysql &>/dev/null        if [ $? -ne 0 ];then                useradd mysql -s /sbin/nologin -M        fi        #useradd mysql -s /sbin/nologin -M        #change datadir owner to mysql        chown -R mysql.mysql $DATADIR        cd        #wget http://mirrors.sohu.com/mysql/MySQL-5.5/mysql-5.5.38.tar.gz        tar xf $VERSION.tar.gz        cd $VERSION        cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/$VERSION \        -D MYSQL_DATADIR=$DATADIR \        -DMYSQL_UNIX_ADDR=$DATADIR/mysql.sock \        -DDEFAULT_CHARSET=utf8 \        -DDEFAULT_COLLATION=utf8_general_ci \        -DENABLED_LOCAL_INFILE=ON \        -DWITH_INNOBASE_STORAGE_ENGINE=1 \        -DWITH_FEDERATED_STORAGE_ENGINE=1 \        -DWITH_BLACKHOLE_STORAGE_ENGINE=1 \        -DWITHOUT_EXAMPLE_STORAGE_ENGINE=1 \        -DWITHOUT_PARTITION_STORAGE_ENGINE=1        make && make install        if [ $? -ne 0 ];then                action "install mysql is failed!"  /bin/false                exit $?        fi        sleep 2        #link        ln -s /usr/local/$VERSION/ /usr/local/mysql        ln -s /usr/local/mysql/bin/* /usr/bin/        #copy config and start file        /bin/cp /usr/local/mysql/support-files/my-small.cnf /etc/my.cnf        cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld        chmod 700 /etc/init.d/mysqld        #init mysql        /usr/local/mysql/scripts/mysql_install_db  --basedir=/usr/local/mysql --datadir=$DAT ADIR --user=mysql        if [ $? -ne 0 ];then                action "install mysql is failed!"  /bin/false                exit $?        fi        #check mysql        /etc/init.d/mysqld start        if [ $? -ne 0 ];then                action "mysql start is failed!"  /bin/false                exit $?        fi        chkconfig --add mysqld        chkconfig mysqld on        /usr/local/mysql/bin/mysql -e "update mysql.user set password=password('$PASSWD') where host='localhost' and user='root';"        /usr/local/mysql/bin/mysql -e "update mysql.user set password=password('$PASSWD') where host='127.0.0.1' and user='root';"        /usr/local/mysql/bin/mysql -e "delete from mysql.user where password='';"        /usr/local/mysql/bin/mysql -e "flush privileges;"        #/usr/local/mysql/bin/mysql -e "select version();" >/dev/null 2>&1        if [ $? -eq 0 ];then                echo "+---------------------------+"                echo "+------mysql安装完成--------+"                echo "+--------- ------------------+"        fi        #/etc/init.d/mysqld stop} install_mysql
编辑配置文件/etc/my.cnf，修改如下
[root@mysql ~]# egrep -v "^#|^$" /etc/my.cnf [client]port		= 3306socket		= /data/mysql/data/mysql.sock[mysqld]port		= 3306socket		= /data/mysql/data/mysql.sockskip-external-lockingkey_buffer_size = 16Kmax_allowed_packet = 1Mtable_open_cache = 4sort_buffer_size = 64Kread_buffer_size = 256Kread_rnd_buffer_size = 256Knet_buffer_length = 2Kthread_stack = 128Kserver-id	= 2            #确保server-id在架构中是唯一的值log-bin=mysql-bin              #二进制日志文件最好打开[mysqldump]quickmax_allowed_packet = 16M[mysql]no-auto-rehash[myisamchk]key_buffer_size = 8Msort_buffer_size = 8M[mysqlhotcopy]interactive-timeout
重新启动mysql
[roo t@mysql ~]# service mysqld restartShutting down MySQL. SUCCESS! Starting MySQL.. SUCCESS! [root@mysql ~]# netstat -anptActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1789/sshd           tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1305/master         tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      12084/mysqld        tcp        0     52 192.168.3.74:22             192.168.3.2:4138            ESTABLISHED 1416/sshd           tcp        0      0 :::22                       :::*                        LISTEN      1789/sshd           tcp        0      0 ::1:25                      :::*                        LISTEN      1305/master
在ha-node1的mysql里建立授权用户rep
mysql> grant replication slave on *.* to 'rep'@'192.168.3.%' identified by 'rep123';Query OK, 0 rows affected (0.00 sec)mysql> flush privileges;Query OK, 0 rows affected (0.00 sec)
查看master状态信息
#在ha-node1上操作mysql> show master status;+------------------+----------+--------------+------------------+| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |+------------------+----------+--------------+------------------+| mysql-bin.000006 |      333 |              |                  |+------------------+----------+--------------+------------------+1 row in set (0.00 sec)
在从库上配置同步参数
mysql> change master to master_host='192.168.3.73',master_port=3306,master_user='rep',master_password='rep123',master_l og_file='mysql-bin.000006',master_log_pos=333;Query OK, 0 rows affected (0.03 sec)#启动slavemysql> start slave;Query OK, 0 rows affected (0.00 sec)#检查状态，状态正常mysql> show slave status\G*************************** 1. row ***************************               Slave_IO_State: Waiting for master to send event                  Master_Host: 192.168.3.73                  Master_User: rep                  Master_Port: 3306                Connect_Retry: 60              Master_Log_File: mysql-bin.000006          Read_Master_Log_Pos: 333               Relay_Log_File: mysql-relay-bin.000002                Relay_Log_Pos: 253        Relay_Master_Log_File: mysql-bin.000006             Slave_IO_Running: Yes            #确保这里是Yes            Slave_SQL_Running: Yes            #确保这里是Yes              Replicate_Do_DB:           Replicate_Ignore_DB:            Replicate_Do_Table:        Replicate_Ignore_Table:       Replicate_Wild_Do_Table:   Replicate_Wild_Ignore_Table:                    Last_Errno: 0                   Last_Error:                  Skip_Counter: 0          Exec_Master_Log_Pos: 333              Relay_Log_Space: 409              Until_Condition: None               Until_Log_File:                 Until_Log_Pos: 0           Master_SSL_Allowed: No           Master_SSL_CA_File:            Master_SSL_CA_Path:               Master_SSL_Cert:             Master_SSL_Cipher:                Master_SSL_Key:         Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No                Last_IO_Errno: 0                Last_IO_Error:                Last_SQL_Errno: 0               Last_SQL_Error:   Replicate_Ignore_Server_Ids:              Master_Server_Id: 11 row in set (0.00 sec)
测试1:现在mysql是运行在ha-node1上，我们创建一个新的数据库rep，查看是否能正常同步
#在ha-node1环境的mysql里创建mysql> create database rep;Query OK, 1 row affected (0.00 sec)mysql> show databases;+--------------------+| Database           |+--------------------+| information_schema || mysql              || performance_schema || rep                |        #创建的新数据库rep| test               |+--------------------+7 rows in set (0.00 sec)#在mysql从服务器上检查结果，结果正常mysql> system hostnamemysqlmysql> system ip a |grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000    inet 192.168.3.74/24 brd 192.168.3.255 scope global eth0mysql> show databases;+--------------------+| Database           |+--------------------+| information_schema || mysql              || performance_schema || rep                || test               |+--------------------+5 rows in set (0.00 sec)
测试2：将ha-node1的heartbeat停止后，所有资源会被ha-node2接管过去，这种情况下mysql从服务器是否还能正常同步
#在ha-node1上停止heartbeat服务，让ha-node2将资源接管过去[root@ha-node1 ~]# /etc/init.d/heartbeat stopSto pping High-Availability services: Done.#在ha-node2上确认资源情况[root@ha-node2 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.7G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot/dev/drbd1            7.8G   48M  7.4G   1% /data[root@ha-node2 ~]# ip a|grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000    inet 192.168.3.72/24 brd 192.168.3.255 scope global eth0    inet 192.168.3.73/24 brd 192.168.3.255 scope global secondary eth0[root@ha-node2 ~]# netstat -anpt |grep mysqltcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      22430/mysqld #在mysql从服务器查看slave状态[root@mysql ~]# mysql -uroot -plyao36843 -h 127.0.0.1 -e "show slave status\G" |grep Yes             Slave_IO_Running: Yes            Slave_SQL_Running: Yes        #这里的结果显示结果正常            #在ha-node2上创建数据库hamysql> create database ha;Query OK, 1 row affected (0.00 sec)mysql> show databases;+--------------------+| Database           |+--------------------+| information_schema || ha                 || mysql              || performance_schema || rep                || test               |+--------------------+8 rows in set (0.04 sec)#在mysql从服务器查看结果[root@mysql ~]# mysql -uroot -plyao36843 -h 127.0.0.1 -e "show databases;"+--------------------+| Database           |+--------------------+| information_schema || ha                 |            #结果正常| mysql              || performance_schema || rep                || test               |+--------------------+
测试3：ha-node1恢复正常后，mysql的从服务器状态是否正常
#重新启动ha-node1的heartbeat服务[root@ha-node1 ~]# /etc/init.d/heartbeat startStarting High-Availability services: INFO:  Resource is stoppedDone.[root@ha-node1 ~]# ip a |grep eth02: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000    inet 192.168.3.71/24 brd 192.168.3.255 scope global eth0    inet 192.168.3.73/24 brd 192.168.3.255 scope global secondary eth0[root@ha-node1 ~]# df -hFilesystem            Size  Used Avail Use% Mounted on/dev/sda3              18G  2.8G   14G  17% /tmpfs                 495M     0  495M   0% /dev/shm/dev/sda1             190M   48M  133M  27% /boot/dev/drbd1            7.8G   48M  7.4G   1% /data[root@ha-node1 ~]# netstat -anpt |grep mysqltcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      24156/mysqld#确认mysql从服务器状态[root@mysql ~]# mysql -uroot -plyao36843 -h 127.0.0.1 -e "show slave status\G" |grep Yes             Slave_IO_Running: Yes            Slave_SQL_Running: Yes#说明在heartbeat+DRBD+mysql的架构中，由于heartbeat主从的切换，mysql从服务器会断开连接60s，然后会自动重连。 
到heart+drbd+mysql主从配置完成


高可用脑裂问题及解决方案
(1)、导致裂脑发生的原因
1、高可用服务器之间心跳链路故障，导致无法相互检查心跳
2、高可用服务器上开启了防火墙，阻挡了心跳检测
3、高可用服务器上网卡地址等信息配置不正常，导致发送心跳失败
4、其他服务配置不当等原因，如心跳方式不同，心跳广播冲突，软件BUG等
(2)、防止裂脑一些方案
1、加冗余线路
2、检测到裂脑时，强行关闭心跳检测(远程关闭主节点，控制电源的电路fence)
3、做好脑裂的监控报警
4、报警后，备节点在接管时设置比较长的时间去接管，给运维人员足够的时间去处理(人为处理)
5、启动磁盘锁，正在服务的一方锁住磁盘，裂脑发生时，让对方完全抢不走"共享磁盘资源"
磁盘锁存在的问题：
使用锁磁盘会有死锁的问题，如果占用共享磁盘的一方不主动"解锁"另一方就永远得不到共享磁盘，假如服务器节点突然死机或崩溃，就不可能执行解锁命令，备节点也就无法接管资源和服务了，有人在HA中设计了智能锁，正在提供服务的一方只在发现心跳全部断开时才会启用磁盘锁，平时就不上锁

(1)、导致裂脑发生的原因

(2)、防止裂脑一些方案

Leave a Comment Cancel reply