Cluster-Computing – Execution of a single periodic task between servers in the cluster

# Attempt to get the lock for'Task X'by setting the current server's hostname
if redis.setnx("lock:task:x" , `hostname`.chomp)
 # Got the lock, now I set it to expire after 5 minutes
 redis.expire("lock:task:x", 60 * 5)
 # This server has the go ahead to execute the task
 execute_task_x
else
 # Failed to get the lock. Another server is doing the work this time around
end

< /p>

(I will try to keep this question as short as possible while clearly describing the situation. Please comment if there are any omissions.)

The situation

> I am running a cluster of three servers in the same data center
>To simplify deployment, each server runs the exact same application code

Goal

>Every minute One server runs a task (call it task X).

Under these conditions

>The cluster is still distributed and highly available
>Every server runs the same application code. In other words, there is no “deploy code A to the main server and code B. Deploy to all secondary servers”.

The reason I don’t want to distinguish between server types is to maintain high availability (to avoid problems when the so-called primary server fails), redundancy (to distribute the load), and To avoid creating a complicated deployment process, I need to deploy different server applications to different types of servers.

Why is it so difficult? If I were to add code that executes this task every 5 minutes, then every server will execute it because every server runs the same application code. Therefore, they need to be able to coordinate which server will run the same during each tick Server.

I can use distributed messaging mechanisms such as Apache Kafka or Redis. If this mechanism is used to coordinate such tasks, how will such “algorithms” work?

I submitted this question to someone else, and his answer was to use the task queue. However, this did not seem to solve the problem, because the question still exists: which server should add tasks to the task queue? If all servers add tasks to the queue, it will result in duplicate entries. Also, which server will execute the next task in the queue? All of these need to be determined through coordination within the cluster, without having to distinguish between different types of servers.

It sounds like you are looking for distributed locks. Redis uses setnx Well done. If you combine it with expire, you can create a global lock that is released every N seconds.

setnx will only write the value, and return true if the key does not exist. Redis operations are atomic operations, so only the first server that calls setnx after the key expires can continue to run the task.

This is an example in ruby:

p>

# Attempt to get the lock for'Task X'by setting the current server's hostname
if redis.setnx("lock:task:x", `hostname`.chomp)
 # Got the lock, now I set it to expire after 5 minutes
 redis.expire("lock:task:x", 60 * 5)
 # This server has the go ahead to execute the task
 execute_task_x
else
 # Failed to get the lock. Another server is doing the work this time around
end

With this, you still rely on calling one Server Redis Master, unless you use redis-sentinel. For information on how to configure automatic failover, please check the redis-sentinel docs.