在 unix 中使用 gnuparallel 时的并发最大进程限制答案

【问题标题】：concurrent max process limit when gnuparallel used in unix在 unix 中使用 gnuparallel 时的并发最大进程限制
【发布时间】：2018-01-02 14:36:33
【问题描述】：

我正在使用gnu parallel 来触发一些并发进程。下面是正在使用的命令。

parallel 'node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES' ::: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ::: 1 2 3 4 5 6 | grep "Query Executed"|wc -l

按照预期，它应该同时触发 120 个并行进程。但是当我执行ps 命令时，我看到只有90 个进程在运行。下面是输出。

[prod PTEST1] parunkarthick@dev118:~> ps -ef|grep node
2951      66965 196290 15 07:50 pts/25   00:00:00 perl /usr/bin/parallel node  /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES ::: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ::: 1 2 3 4 5 6
2951      67082  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 1
2951      67087  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 2
2951      67088  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 3
2951      67093  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 4
2951      67098  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 5
2951      67103  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 6
2951      67108  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 1
2951      67113  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 2
2951      67118  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 3
2951      67123  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 4
2951      67128  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 5
2951      67133  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 6
2951      67138  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 1
2951      67143  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 2
2951      67148  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 3
2951      67153  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 4
2951      67158  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 5
2951      67163  66965 25 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 6
2951      67168  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 1
2951      67173  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 2
2951      67178  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 3
2951      67183  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 4
2951      67188  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 5
2951      67193  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 6
2951      67198  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 1
2951      67203  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 2
2951      67208  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 3
2951      67213  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 4
2951      67218  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 5
2951      67223  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 6
2951      67228  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 1
2951      67233  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 2
2951      67238  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 3
2951      67243  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 4
2951      67248  66965 25 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 5
2951      67253  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 6
2951      67258  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 1
2951      67263  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 2
2951      67268  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 3
2951      67273  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 4
2951      67278  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 5
2951      67283  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 6
2951      67288  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 1
2951      67293  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 2
2951      67298  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 3
2951      67303  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 4
2951      67308  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 5
2951      67313  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 6
2951      67318  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 1
2951      67323  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 2
2951      67328  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 3
2951      67333  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 4
2951      67338  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 5
2951      67343  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 6
2951      67348  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 1
2951      67353  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 2
2951      67358  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 3
2951      67363  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 4
2951      67368  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 5
2951      67373  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 6
2951      67378  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 1
2951      67383  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 2
2951      67388  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 3
2951      67393  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 4
2951      67398  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 5
2951      67403  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 6
2951      67404  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 1
2951      67409  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 2
2951      67418  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 3
2951      67423  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 4
2951      67428  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 5
2951      67433  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 6
2951      67438  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 1
2951      67443  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 2
2951      67448  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 3
2951      67453  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 4
2951      67458  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 5
2951      67463  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 6
2951      67468  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 1
2951      67473  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 2
2951      67478  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 3
2951      67483  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 4
2951      67488  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 5
2951      67493  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 6
2951      67498  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 1
2951      67503  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 2
2951      67508  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 3
2951      67513  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 4
2951      67523  66905  0 07:50 pts/0    00:00:00 grep node
[prod PTEST1] parunkarthick@dev118:~> ps -ef|grep node|wc -l
90

为什么只有 90 个进程同时被触发？

当我检查 ulimt -u 时，它的限制为 2000，所以真的无法理解为什么它只产生了 90 个并行进程。

[prod PTEST1] parunkarthick@dev118:~> ulimit -u
2000

任何帮助将不胜感激！！！

【问题讨论】：

你的 CPU 有 90 个内核吗？试试parallel --number-of-cores，它会告诉你默认的并行度。
是的。有没有办法增加？
你有多少核心和多少内存？
它显示。 cat /proc/cpuinfo|grep cores|wc -l 88

标签： unix gnu-parallel ulimit

【解决方案1】：

默认情况下，GNU Parallel 将使用与您拥有的 CPU 内核数量相匹配的并行度，您可以使用它进行测试：

parallel --number-of-cores

如果您想运行不同数量的并行作业，请使用 -j 参数，例如对于 200 个并行作业，使用：

parallel -j 200 ...

或者，如果您希望并行作业比您拥有的内核多 50% 左右，请使用百分比：

parallel -j 150% ...

顺便说一句，如果您使用的是bash，那么使用以下语法，您的命令可能会稍微不那么笨拙：

parallel  'node  /home/XYZ.js F_CHARGES' ::: {1..20} ::: {1..6}

【讨论】：

当我测试非常大的数字时收到这样的警告。 parallel -j 200 'node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES' ::: {1..20} ::: {1..10} ::: {1..5} |grep Connected|wc -l parallel: Warning: No more processes: Decreasing number of running jobs to 199. Raising ulimit -u or /etc/security/limits.conf may help. parallel: Warning: No more processes: Decreasing number of running jobs to 198. Raising ulimit -u or /etc/security/limits.conf may help.
我的 ulimit 为 1024，但只能触发大约 300 个进程
你没有提到你的操作系统，但是看看这里...gnu.org/software/parallel/…
它是 Linux x86_64
会不会是nodescript_pd.js 产生了额外的进程？这行得通吗：parallel -j200 sleep ::: {100..300}