Shell - small-scale load balancing - balance, load, shell, small scale

I have a series of tasks that need to be completed; there are no dependencies between the tasks. I am looking for a tool that can help me distribute these tasks to machines. The only restriction is that each machine Should only run one job at a time. I am trying to maximize throughput because the work is not very balanced. My current shell script that is being hacked is not efficient because I have pre-built job queues for each machine and cannot reload the jobs. The load machines are moved from the queue to the machines waiting for all operations to be completed.

The previous suggestions include SLURM, which seems a bit overkill, and even more of an overkill LoadLeveller.

GNU Parallel looks almost what I want, but the remote machine does not speak SSH; there is a custom job launcher (without queuing function). What I want is Gnu Parallel, before the job is scheduled, the machine can Replace it directly with a shell script.

So, to summarize:

>List of acceptable machines for the work list: maximize throughput. As close as possible to the shell.

The worst case might be attacked with bash’s lockfile, but I feel as if a better solution must exist somewhere.

Suppose your job looks like in the text file jobs.tab

/path/to/job1< br />/path/to/job2
...

Create dispatcher.sh like this

mkfifo /tmp/jobs.fifo 
while true; do
 read JOB
 if test -z "$JOB"; then 
 break 
 fi
 echo -n "Dispatching job $JOB .."
 echo $JOB >> /tmp/jobs.fifo
 echo ".. taken!"
done
rm /tmp/jobs.fifo

< p>and run an instance

dispatcher.sh 
Now create launcher.sh as

while true; do
 read JOB  if test -z "$JOB"; then
 break
 fi

 #launch job $JOB on machine $0 from your custom launcher

done And run an instance of launcher.sh for each target machine (using the machine as the first and only parameter)

I have a series of things that need to be completed Work; there are no dependencies between jobs. I'm looking for a tool that can help me distribute these jobs to machines. The only limitation is that each machine should only run one job at a time. I'm trying to maximize throughput because of work Not very balanced. My current hacked shell script is not efficient because I pre-built the job queue of each machine and cannot move the job from the queue of the heavy load machine to the machine waiting for all operations to be completed.

The previous suggestions include SLURM, which seems a bit overkill, and even more of an overkilling LoadLeveller.

GNU Parallel looks almost what I want, but the remote machine Don't talk about SSH; there is a custom job launcher (without queuing function). What I want is Gnu Parallel. Before scheduling the job, the machine can be directly replaced with a shell script.

So, to summarize :

>List of acceptable machines for the work list: maximize throughput. As close as possible to the shell.

The worst case may be attacked together with bash's lockfile, but I It feels as if a better solution must exist somewhere.

Assuming your job looks like in the text file jobs.tab

/path/to/job1
/path/to/job2
...

Create dispatcher.sh like this

mkfifo /tmp/jobs.fifo
while true; do
 read JOB
 if test -z "$JOB"; then 
 break 
 fi
 echo -n "Dispatching job $JOB .."
 echo $JOB >> /tmp/jobs.fifo
 echo ".. taken! "
done
rm /tmp/jobs.fifo

and run an instance

dispatcher.sh  Now create launcher.sh as
 
 while true; do
 read JOB  if test- z "$JOB"; then
 break
 fi

 #launch job $JOB on machine $0 from your custom launcher

done
And run a launcher.sh instance for each target machine (using the machine as the first and only parameter)

Shell – small-scale load balancing

Leave a Comment Cancel reply