Github has been focusing on providing news push services for developers for many years. Through a push SDK, the mobile terminal establishes a long connection with the server and maintains an online state. However, in the case of network abnormalities, the message cannot be delivered to the end user in real time, so the push server establishes an offline message list to wait for the user to log in again to deliver the message. This part of data is stored in a push Redis cluster. The entire cluster includes more than 100 instances of master and slave. The number of keys is at the level of 1 billion, and the storage space is at level T, which brings certain maintenance costs and operation and maintenance challenges. As a push back-end development engineer, we are always looking for cost-effective solutions.
Push the QPS of the entire cluster at the million level, if you choose to use AerospikeCompared with the actual measurement, we found that a single physical machine equipped with a single Inter SSD 4600 can achieve a QPS close to 10w, that is, dozens of machines can meet the existing needs and support the business for a long period of time in the future need.
The advantages of Aerospike
Aerospike is a high-performance, scalable, and reliable NoSQL solution. It supports RAM and SSD as storage media, and is specially optimized for SSD. It is widely used in real-time bidding and other real-time computing fields. The official guarantee is that 99% of operations are completed within 1ms, and it provides functions such as automatic cluster data rebalance, cluster-aware client, and supports the storage of very large-scale data sets (100T level).
As a KV storage, Aerospike provides a variety of data types, and its operation is similar to Redis. In addition to basic functions, Aerospike also supports multiple monitoring methods such as AMC console, API, and multiple monitoring indicators such as cluster QPS, health, and load, which are more friendly to operation and maintenance. Supports automatic Rebalance of data in the cluster. Compared with the Redis cluster solution, the maintenance cost is reduced a lot.
This article mainly does some experience sharing of Aerospike gray-scale deployment and use, and hopes to provide some references for readers who are investigating or are ready to use Aerospike. In addition, the concept of gray scale is not limited to Aerospike itself. The migration and planning of other basic components can also bring certain reference significance.
Data model description
< img alt="share picture" class="" src="/wp-content/uploads/images/database/nosql/1626816674242.jpg
Aerospike uses modeless storage, The data model is similar to RDBMS, so it is relatively friendly in understanding and use:
Each namespace contains multiple sets, each set contains multiple records, and each record contains multiple bins (database columns), which can be accessed by index key Query record. Different services can use different namespaces of the same cluster for resource isolation, so as to achieve resource pooling and maximize resource utilization.
Gray scale online process
Tweet offline messages The list storage business uses a large-scale Redis cluster. We have investigated disk storage that supports the Redis protocol, such as ssdb and pika. Based on the overall calculation, Aerospike is the most cost-effective.
In the early stage, we combined the online scene to simulate the actual reading and writing ratio (analysis online For business, we found that the approximate ratio of writing and reading is between 1:1 and 1:2) to perform stress testing, evaluate and verify the feasibility, and then plan for production.
Online business is relatively complex, and it is not realistic to switch to Aerospike directly, and the risk is relatively high. It is difficult for the test network simulation verification to expose the problems that may occur in the production environment, so we divide the entire online process into the observation phase and the gray phase. In the observation phase, as the name suggests, the original Redis cluster still undertakes online read and write services, but only copies the same traffic into Aerospike for real pressure verification; in the gray-scale phase, the online business is gradually switched to the Aerospike cluster to expand the gray-scale guarantee cluster Stable operation until the business is completely cut to Aerospike. The specific operations of the two stages are as follows:
Observation phase: Redis After the operation is successful, the read and write operations on Redis are synchronized to Aerospike in an asynchronous manner, and Aerospike does not undertake specific business. The next step is to write data to Redis and Aerospike. At this stage, the main observation is whether the data on both sides are consistent, the Aerospike pressure and so on. At the same time, during the observation phase, operation and maintenance operations such as node restart and cluster expansion can be performed to evaluate the operation and maintenance cost, and optimize the configuration. Here, you can use the AMC page console and monitoring API to monitor the cluster status, and the client calls part to record necessary logs and monitoring information.
Gray stage: Aerospike began to assume part of the application and task offline message list storage. Gray stage Redis and Aerospike Data double write and double clear, keep hot standby state until Redis data is completely switched to Aerospike and run stably for a period of time.
The observation phase is very important. It is basically an online assessment of the feasibility of the entire program. The observation point at this stage is divided into two parts: the client (AS-Client) and the server (AS-Server). Client main observation:
1. Use metrics to monitor client request response time, and use the percentage distribution of request time consumption over a period of time (50%, 90%, 99%) , 99.9%), evaluate the system SLA.
2. Monitor the count of read and write success and failure.
3. Set the slow log threshold to 50ms, and count the slow log conditions during peak periods and normal periods.
4. Write Aerospike queue monitoring asynchronously, and adjust the queue size reasonably.
Main observation on the server side:
1. The health of the cluster.
2. Disk and memory usage, memory space/disk space ratio.
3. Machine IO load, CPU load, disk fragmentation and other information.
4. Cluster throughput, whether the read and write TPS can be equivalent to the online Redis cluster.
5. Data consistency check. How to check the consistency of the two data in the observation phase and the gray phase? The key-by-key comparison difference is difficult to meet the requirements in terms of performance. Considering that the data is completely consistent, the data detected by Redis should be exactly the same as the data detected by Aerospike, so the data query results of Redis and Aerospike are sampled and recorded in the log, and the comparison and analysis are inconsistent within 1 minute, 5 minutes, 30 minutes, and 1 hour. Percentage of data. If the number of online keys is at the level of 1 billion, even if only one-tenth of a ten thousand difference is detected, the inconsistency is significant. In this case, it is necessary to investigate and resolve the cause of the inconsistency.
Maintainability mainly considers that automatic rebalance of cluster data will bring certain performance degradation, which may have a greater impact on user experience. Based on our experience, we simulated some typical Operation and maintenance scenarios:
1. Simulate the impact of cluster Rebalance on system performance caused by a single node failure.
2. Simulate the impact of cluster Rebalance on system performance caused by cluster expansion.
3. According to the impact on online business, calculate the rebalance speed of the cluster during the day and night, and support cron job update.
4. The node restarts.
5. Add SSD mount.
6. Optimization of related configuration, etc.
To sum up, the complete online process is divided into the following steps:
0. Simulate the online environment pressure test for feasibility verification.
1. Put the Aerospike client Encapsulate it into a Redis-like interface, add necessary logs, monitoring items, and check the validity of Bin.
2. The message service integrates the Aerospike client. The required functions include: Aerospike asynchronous read and write, business data source switching, traffic filtering, etc.
3. QA function verification.
4. Apply for resources and deploy Aerospike cluster online.
5. The messaging service integrated with Aerospike is launched.
6. After passing the verification in the observation phase, it will enter the gray phase until it is finally online or withdrawn halfway.
Summary of experience
During the use of Aerospike, we encountered some problems and challenges, which can be summarized as follows:
1. Aerospike enables single-bin mode to save space.
2. Aerospike does not store the original key. The actual index is a 20-byte hash value of the original key. If the business needs to use the original key, the bin storage must be set separately. Even if the number of bytes of the key and value values is small, the key itself occupies 20 bytes, so the actual space occupied will be relatively large.
3. Aerospike will rebalance data when the node is down or adding or removing nodes. This process will affect the quality of external services. However, the speed of Rebalance can be controlled, so it is necessary to make a trade-off between ensuring the quality of service and the rapid recovery of the cluster.
4. The community version cluster has to rebuild the index every time it restarts, and then load it into the memory, which will result in a slower speed. The namespace needs to be specified in the configuration file, so it is best to divide it by business and allocate the namespaces that may be used in the future to reduce unnecessary restarts.
5. Because the SSD itself has fragmentation and write amplification problems, in actual use, we found that if the disk space usage is about 50%, the performance degradation will be more serious. Therefore, the relevant parameters of defragmentation can be optimized in combination with the actual business.
6.Aerospike has restrictions on HotKey, so when a key is frequently read and written, a HotKey error (errorcode 14) will be returned. The server can increase the transaction-pending-limit configuration to increase the amount of concurrency for the same key operation. The default is 20, and the value 0 means unlimited. Increasing this configuration may degrade certain performance. The client may need to add retry processing to the exception, but retrying may further increase the risk of HotKey.
7. The replacement of this basic component must use online traffic as much as possible to do stress testing, so as to expose potential problems as soon as possible.
8. During the observation phase, the operation and maintenance costs should also be evaluated to avoid jumping from one pit to another.
9. In the process of using, you also need to pay attention to some inherent limitations of Aerospike, such as a namespace can have up to 1023 sets, the bin name length can be up to 14 single-byte characters, and a namespace Supports up to 64 SSDs, etc., for details, please refer to: aerospike_known_limitations.
Conclusion
Aerospike, as a large-capacity NoSql solution, has not been widely used in domestic factories. It is suitable for scenarios with relatively large capacity requirements and relatively low QPS, which can save TCO to a certain extent. In terms of supporting commands, Aerospike and Redis are more similar, and business migration is easier. It naturally supports cluster deployment and is more friendly to monitoring and operation and maintenance support. Despite having so many excellent features, it is still necessary to be cautious when selecting technology, and assess in advance whether it meets your own business scenarios, whether performance and cost can meet the requirements, etc. In some official test scenarios, its performance is higher than Redis. In fact, due to the limitations of SSD itself, in most cases, it has a large gap with Redis in terms of QPS. Finally, you must go through online traffic verification before going online, and handle the actual online business in a grayscale manner to minimize the impact on the user experience.