Shell – small-scale load balancing

/path/to/job1< br />/path/to/job2
...

mkfifo /tmp/jobs.fifo 
while true; do
 read JOB
 if test -z "$JOB"; then 
 break 
 fi
 echo -n "Dispatching job $JOB .."
 echo $JOB >> /tmp/jobs.fifo
 echo ".. taken!"
done
rm /tmp/jobs.fifo

dispatcher.sh 
Now create launcher.sh as

while true; do
 read JOB  if test -z "$JOB"; then
 break
 fi

 #launch job $JOB on machine $0 from your custom launcher

done 
And run an instance of launcher.sh for each target machine (using the machine as the first and only parameter)
 

I have a series of things that need to be completed Work; there are no dependencies between jobs. I'm looking for a tool that can help me distribute these jobs to machines. The only limitation is that each machine should only run one job at a time. I'm trying to maximize throughput because of work Not very balanced. My current hacked shell script is not efficient because I pre-built the job queue of each machine and cannot move the job from the queue of the heavy load machine to the machine waiting for all operations to be completed.

The previous suggestions include SLURM, which seems a bit overkill, and even more of an overkilling LoadLeveller.

GNU Parallel looks almost what I want, but the remote machine Don't talk about SSH; there is a custom job launcher (without queuing function). What I want is Gnu Parallel. Before scheduling the job, the machine can be directly replaced with a shell script.

So, to summarize :

>List of acceptable machines for the work list: maximize throughput. As close as possible to the shell.

The worst case may be attacked together with bash's lockfile, but I It feels as if a better solution must exist somewhere.

Assuming your job looks like in the text file jobs.tab

/path/to/job1
/path/to/job2
...

Create dispatcher.sh like this

mkfifo /tmp/jobs.fifo
while true; do
 read JOB
 if test -z "$JOB"; then 
 break 
 fi
 echo -n "Dispatching job $JOB .."
 echo $JOB >> /tmp/jobs.fifo
 echo ".. taken! "
done
rm /tmp/jobs.fifo

and run an instance

dispatcher.sh  
Now create launcher.sh as
 
 
while true; do
 read JOB  if test- z "$JOB"; then
 break
 fi

 #launch job $JOB on machine $0 from your custom launcher

done
And run a launcher.sh instance for each target machine (using the machine as the first and only parameter)