Background

Many businesses of the company are using RocketMQ. The business volume in the early stage is small and there is no problem, and there is not too much attention to the availability of the cluster. Therefore, the business of the whole company shares a cluster, and the business volume of the company increases. Business relies more and more on RocketMQ clusters, and I began to think about cluster splitting and risk sharing; I started thinking about division by department, one cluster for each department, but there is a need for data interaction between departments through RocketMQ, which will bring about The problem is that an application needs to connect to two RocketMQ clusters, and the business side needs to change the code. How can the cluster be achieved without a single point, risk sharing, and the cluster can be horizontally expanded in the future? Our solution As follows:

Cluster deployment planning

The entire cluster is divided into three layers, namely the application access layer, the nameserver cluster and the broker cluster. The three parts are explained below:

Access layer

The access layer is actually the address where the application connects to the MQ cluster. In the current production environment, we are directly connected to the IP address of the nameserver. If the nameserver is expanded or the server is changed, you need to modify the configuration Restart the service to update the new nameserver address. Although the probability of this event is relatively low, it is still more troublesome if it happens. All our new access solutions are load balancing + domain name. You configure the MQ address in the program as: BLB: 9876,nameserver1.chj.cloud:9876,nameserver2.chj.cloud:9876, so that both load balancing and domain name insurance are done at the same time. If the load balancer fails, the application will retry to connect to the domain name

Nameserver cluster

The nameserver itself is stateless and there is basically no pressure, so we plan to deploy two in the early stage; when the capacity needs to be expanded later, the brokers need to be restarted in turn, so that the brokers are also registered on the new nameserver

Broker cluster

Broker is responsible for the storage of messages. There are multiple modes of Broker deployment. We will deploy two architectures according to business needs: multiple masters and multiple slaves (synchronous replication, synchronization Brush), multiple masters and multiple slaves (asynchronous replication, asynchronous brushing); clusters can also be expanded horizontally

Broker naming convention: odd numbers are asynchronous replication + asynchronous brushing type, even numbers are synchronous replication + synchronous brushing Disk type, such as: broker-1, broker-2

Architecture diagram

Topic Management

According to the business scenario, we create Topic on different Broker clusters. The business scenarios are divided into two situations : One is to ensure that the message is not lost, but the throughput requirements are not high, such as order-related information. For example, the TopicA in the figure below is used to store the topic related to the order, then we will create this topic for synchronous replication and synchronous brushing. On a Broker cluster with such a role as the disk; the other is that the throughput requirements are particularly high, but a small amount of message loss is allowed, such as the monitoring related information reported by the vehicle. For example, TopicB in the figure below is used to store the monitoring related information reported by the vehicle Topic, then we will create this topic on the Broker cluster with asynchronous replication and asynchronous flushing