一. Background
ibmmq is a traditional mq product, which is stable in operation and has its own advantages. However, in the area of high availability (HA), the company needs to use high availability (HA) products according to its own needs. However, due to the market HA Commercial products are more expensive, so it is the cheapest and most suitable to use heartbeat products at the Linux operating system level. There are many options for ibmmq network shared storage, the closest to the production environment is the storage of ISCSI protocol, so use ISCSI as the shared storage of mq server (there is a little storage knowledge below).
Deployment architecture diagram:
2. Solutions
1, build a shared storage environment; span>
2. Install heartbeat;
3. Install mq software;
4 , Complete the relevant scripts;
5. Test
< span style="font-family:'Microsoft YaHei'; font-size: 14pt; color: #ff0000;">Three, use technology stack and version
1, linux version: redhat6.7
2. Shared storage, iscsi
Target version: scsi-target-utils -1.0.24-18.el6.x86_64
Initiator version ：Iscsi-initiator-utils-6.2.0.873-27.el6_9.x86_64
Network shared storage introduction: According to the network structure: Literacy article: http://www.ptyqm.com/19026.html
According to the production environment, only two shared storage implementations are considered
1) NAS: Network attached storage, NFS shared storage is simple to implement and convenient to use. It is used in the form of a file system because it is a file It is provided in a systematic way, so the data transmission speed is relatively slow;
2) IP-SAN: IP storage is realized through the TCP/IP protocol. The main protocols currently realized are: iscsi (Internet Small Computer System Interface), which can be realized in a virtual machine and is close to the production environment.
iscsi: a file block level protocol, transmission speed Faster, share storage devices via tcp/ip. Local disks can be made into lvm (logical volume management), which is a management mechanism for disk partitions in Linux. A logical layer is added between the disk partition and the file system to shield the disk partition layout for the file system. The advantage is that the logical volume can be adjusted at any time to increase or decrease the space.
iscsi is divided into target and initiator in use. The target end is the storage end, and the initiator end is the network storage end.
target and initiator can be installed separately. Once installed, it can be used through configuration and commands. The convenience is that the network storage seen on the initiator side is only equivalent to a local disk, such as /dev/sdb, and the rest can be used after formatting.
Reference translation: https://linux.cn/article -4971-1.html
The next few articles are very A detailed introduction to the use of iscsi, the original English is at: https://www.tecmint.com/create-centralized-secure-storage-using-iscsi-targetin-linux/
< span style="font-family:'Microsoft YaHei'; font-size: 16px; color: #000000;"> are good conscience articles.
3. High-availability software heartbeat p>
heartbeat software is provided by Linux, but it needs to be installed separately. It can be installed on the node where there is a demand.
This software can connect a certain network card, such as eth0, virtual An ip is provided to the outside world. This vip is also a resource in heartbeat and requires master and salve to use each other.
This software can pass the heartbeat cable (another network card such as eth1 ) Send a heartbeat packet with your own salve or master, you have checked whether the other party is active, the heartbeat interval and the dead time are both Can be configured in the configuration file.
This software can also do ha without heartbeat, But the disadvantage lies in the possibility of split brain phenomenon, especially in the case of poor network conditions.
Note here that the heartbeat packet is sent through the udp protocol , The surprise is that heartbeat not only has a nice name, but also can realize unicast, multicast and broadcast. In this way, the cluster ha solution can be realized in this way. Mark here first, and then research if necessary.
There are many pitfalls in the use of heartbeat, which will be mentioned below .
Recommend good articles: https://www.cnblogs.com /zhanjindong/p/3618055.html
Version: heartbeat- 3.0.4-2.el6.x86_64
of course during installation There is more than one package, other packages will be mentioned below.
Four. Difficulties span>
1, it is difficult to build iscsi, Not familiar with Linux disk and storage relationship and commands;
2. The installation software lacks a serious dependency, and the production environment is an intranet environment. All rpm download is complete to install;
3. I am not familiar with the principle of heartbeat, I don’t know how VIP is used in it, and I don’t know Its switching process;
4. Switch between the main and backup software of heartbeat When changing, I am not familiar with the writing of the script.
5. Time point Estimated
1. One day to study iscsi storage and its implementation and basic principles, and build its environment, the test is completed (focus on use);
2. Study the heartbeat installation process in one day , Understand its principle, install mq software to try configuration;
3. Test heartbeat one day, observe its abnormal phenomenon and analyze and solve it;
4. Summarize the implementation process , Prepare the installation package for the launch and sort out the process.
VI. Virtual machine implementation
1. Configure the target server of iscsi on the LVM server to configure lvm and configure the configuration file of the target server to publish the local lvm to the network; the shared storage size is configured to 10G;
Create locally Good lvm:
Target server/etc/tgt/targets.conf configuration span>< /p>
< /span>
< p> Check again the external release situation of the iscsi target server
p>
2. Install the initiator of iscsi on the MQMaster and MQSalve servers and find the shared lvm of the target service. The pitfall here is that the authentication of the target must be complete.
After logging in to the target server, restart the linux system again in the future, you can log in to the target directly.
After logging in, you can see a new disk on the local disk:
After that, you only need to create a new partition for sdb, and then format the newly created partition. You can mount the storage to any local directory, so that it is consistent with the normal file system for use.
For MQSalve, because the partition creation of sdb is performed on MQMaster, if iscsi is used on MQSalve, it is necessary to format the sdb and then mount.
After many experiments here, it is best not to mount the shared network storage on two active and standby nodes at the same time. At present, data errors may be found, and the entire file system may even fail to work normally. condition.
3. Install mq software in MQMaster and MQSalve respectively;
4. Install the heartbeat software on MQMaster and MQSalve;
About installing heartbeat, because the production environment is considered as an intranet environment and there is no Internet access, then the dependency problem of installing the rpm package has become a big problem, which is a huge pit.
At present, it is adopted to download all the rpm packages to the local through yum method, and then take it out to install the entire directory rpm. The tips used here are:
< span style="font-size: 16px; color: #000000;">The heartbeat software is divided into three small projects. It is not possible to download directly from yum.
yum install yum-plugin-downloadonly
In this way, all the rpms are downloaded, and then they can be directly taken to the official environment for installation. If there is a shortage of associations, you can download other relevant rpms in the same way. The method is always more convenient than looking for rpm one by one, especially when there is no operation and maintenance.
Installation:
rpm -ivh ./rpms/*
5. Configure the heartbeat software and step on the pit.
After installing the heartbeat software, it will create a directory under /etc by default: ha.d
At this time, you must set 3 configuration files: as shown in the red picture above Box, cp come over when you find it. . .
Tiankeng 1: The permission of the authkeys file can only be 600, otherwise heartbeat will keep reporting errors. I don’t know if it is a heartbeat version problem or something else. . .
Tiankeng 2: The scripts written by yourself must be placed in the resource.d directory. After you enter, you will find many service-related scripts. There is no official prompt for this, so I looked for it for a long time. . .
Regarding the configuration of the authkeys file, pull a lot on the Internet, and drag one. The master and the salve can be used if they are the same, but it must be: 
About the configuration of the haresource file, ha, the resource file of ha: The file provides a lot of configuration instructions, take a screenshot:
According to the translation, separate according to the space, first The domain name that provides the service is a small pit. Note that the domain name configuration must be added to the hosts, otherwise it will not take effect. . . Report an error. . . The second is that the service IP address can have multiple, you can specify the network card and the default subnet mask, you can broadcast the address, and finally the script to be executed. . .
My configuration: MQMaster 192.168.0.208/24/eth0 mq, the first is the name of the master node, the second is the vip and subnet mask and the designated vip network card, and the last is my script mq, This configuration of master and slave nodes must be consistent, but in the official guidance, there is a sentence:
#They must match the names of the nodes listed in ha.cf, which in turn
# must match the `uname -n` of some node in the cluster. So they aren’t
# virtual in any sense of the word.
The hostname of the master node must be in another configuration file ha It appears in .cf and the specified attribute item is: node MQMaster, otherwise VIP will fail~
Heartbeat is used here. Note that there is no need to create VIP first. Heartbeat will respond according to the resource configuration in haresource. The creation of the VIP of the network card, in the recognition of the heartbeat software, VIP is an extremely important resource: After starting the heartbeat master: 
In the following text, you can also see the heartbeat startup process in the log.
About the configuration of ha.cf:
This is relatively simple. After the event, you can operate the service through service heartbeat stop/restart/start. This service starts with the operating system, but according to the test, The pitfall is that sometimes only the master machine is started, it does not take resources, but the heartbeat service is running, and you need to restart the service manually. . .
6. Analyze the heartbeat log file.
The process of completing a switch is roughly as shown in the log, and it becomes the same if MQMaster is replaced with MQSalve.
7. Test the heartbeat and get the test result.
I won’t explain it much, the only thing I found is that split brain often occurs when the network environment is not good. . . The result of the evil strategy is to damage network storage. . . Therefore, the heartbeat line must be broken. . . And it must be in-line. . . You can’t use the switch, it’s not reliable, otherwise, once the storage is damaged, then the application software will have to be reinstalled. The matter is small, the point is. . . The data is gone. . . Therefore, raid10 is also very important for shared storage. . .
Seven. Thanks
Thanks to two colleagues from outside the company for their previous exploration and provided the heartbeat solutions and mq software scripts.
Thanks for the good articles in the article and some online articles, which benefited a lot.