如何等到在一个while循环中启动的所有后台作业都完成了某个步骤？

Question

我想在 bash 脚本的 while 循环中启动一系列后台作业：

END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

在这个while循环之外，我想等到所有这10个后台作业都完成SOME COMMAND 1然后继续，所以伪代码类似于：

until [the number of process that finished SOME COMMAND 1 is greater or equal to $END];
do
sleep 0.5
done

我试图做的是添加另一个这样的计数器：

true_started_cnt=0
END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
true_started_cnt=`expr $true_started_cnt + 1`;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

until [ $true_started_cnt -ge $END ]
do
echo "Waiting for initial period for all streams to finish"
sleep 0.5
done

但这似乎不起作用，可能是因为多个 bg 作业同时写入同一个全局变量。我想知道如何在这段代码中实现我的意图。

参考this post，我也在尝试：

echo 1 >/dev/shm/foo

END=10
i=0;
while [ $i -lt $END ]
do
(SOME COMMAND;\
sleep 3;
SOME COMMAND 1;\
echo $(($(</dev/shm/foo)+1)) >/dev/shm/foo;\
SOME COMMAND 2;\
SOME COMMAND 3;) &
i=`expr $i + 1`
done

until [ $(echo $(</dev/shm/foo)) -ge $END ]
do
  echo "Waiting, true_started_cnt=$(echo $(</dev/shm/foo))"
  sleep 1
done

但还是不行。

Answer 1

您的上一个脚本中也可能存在竞争条件，试试这个：

cp /dev/null /dev/shm/foo

END=10
for ((i=0; i<$END; i++)); do
    (
    echo SOME COMMAND;\
    sleep $((i+1));\
    echo $i SOME COMMAND 1;\
    echo $i >>/dev/shm/foo;\
    echo $i SOME COMMAND 2;\
    sleep $((i+1));\
    echo $i SOME COMMAND 3
    ) &
done

until [ $(wc -l /dev/shm/foo | awk '{print $1}') -ge $END ]
do
    echo "Waiting, /dev/shm/foo contains $(echo $(</dev/shm/foo))"
    sleep 1
done

echo "*********** All SOME COMMAND 1 finished"
wait
echo "Script finished"

Answer 2

如果您只想确保所有进程都退出，则不需要记录进度。

#!/bin/bash

END=10
pids=()

for (( i = 0; i < END; ++i )); do
    ( sleep "$(( RANDOM % 10 + 1 ))" ) &
    pids[$!]=$!
done


while [[ ${#pids[@]} -gt 0 ]]; do
    for pid in "${pids[@]}"; do
        echo "Waiting for $pid to exit."
        wait "$pid"
        kill -s 0 "$pid" >/dev/null || unset 'pids[pid]'
    done

    sleep 1
done

Answer 3

您可以使用 FIFO 队列来表示某个阶段的作业完成：

会合.sh

#!/usr/bin/env bash

shopt -so errexit
shopt -so nounset

declare -ri job_count=5

# A FIFO queue where jobs will signal their completion.
mkfifo rendezvous

# Clean up FIFO queue on script exit.
trap EXIT EXIT; EXIT() {
  rm -f rendezvous
  trap  - EXIT      # Restore default interrupt handler.
  kill -s EXIT "$$" # Propagate interrupt.
}

# Wait until the known number of jobs have signaled their completion to the
# FIFO queue.
wait_rendezvous() {
  (
    local -i counter=0

    while read -r; do
      (( ++counter ))

      if (( counter == job_count )); then
        break
      fi
    done
  ) < rendezvous
}

# Signal completion to the FIFO queue.
show_at_rendezvous() {
  # For some reason, without the trailing ampersand some A jobs end up
  # dangling.
  echo "done" > rendezvous &
}

#
# Actual script logic starts below.
#
command_a() {
  local -i job_id=$1
  echo "Command A[$job_id]: start"
  sleep 5
  show_at_rendezvous
  echo "Command A[$job_id]: done"
}

command_b() {
  local -i job_id=$1
  echo "Command B[$job_id]: start"
  sleep 10
  echo "Command B[$job_id]: done"
}

# Launch jobs.
for i in $(seq 1 "$job_count"); do
  command_a "$i" &
  command_b "$i" &
done

# Wait for all A jobs to signal their completion.
wait_rendezvous
echo "All A commands done; now awaiting for B commands..."

# Wait for all remaining jobs to finish.
wait
echo "All B commands done."

输出

$ ./rendezvous.sh 
Command A[1]: start
Command B[1]: start
Command A[2]: start
Command B[2]: start
Command A[3]: start
Command B[3]: start
Command A[4]: start
Command B[4]: start
Command A[5]: start
Command B[5]: start
Command A[2]: done
Command A[1]: done
All A commands done; now awaiting for B commands...
Command A[5]: done
Command A[3]: done
Command A[4]: done
Command B[1]: done
Command B[2]: done
Command B[3]: done
Command B[4]: done
Command B[5]: done
All B commands done.