shell：参数列表太长答案

【问题标题】：shell: Argument list too longshell：参数列表太长
【发布时间】：2021-10-26 01:52:36
【问题描述】：

我想同时对具有多个请求的端点进行负载测试。我收到错误Argument list too long。我该如何解决这个问题？

import subprocess
import argparse
def main(**kwargs):
    my_parser = argparse.ArgumentParser()
    my_parser.add_argument('-s')
    my_parser.add_argument('-e')
    args = my_parser.parse_args()
    i = int(vars(args)['s'])
    cmd = ''
    f= open('test.txt')
    while i <= int(vars(args)['e']):
        for line in f:
            cmd += f'curl -X POST -H "Content-Type: application/json" https://some_api_ep -d {line} &'
        i +=1
    subprocess.run(cmd, shell=True)
if __name__ == "__main__":
    main()

$ python load_generator.py -s 1 -e 1
Traceback (most recent call last):
  File "load_generator.py", line 20, in <module>
    main()
  File "load_generator.py", line 17, in main
    subprocess.run(cmd, shell=True)
  File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: '/bin/sh'

总记录数

$ cat test.txt | wc -l
18732

样本数据

$ cat test.txt
'{"ErrorCode": 55201, "ErrorClass": 8, "NodeID": 32, "Params0": 1, "Params1": 70451202, "Params2": 9530, "Params3": 7, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}' 
'{"ErrorCode": 55004, "ErrorClass": 8, "NodeID": 32, "Params0": 538990697, "Params1": 410814, "Params2": 410931, "Params3": 0, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}' 
'{"ErrorCode": 282, "ErrorClass": 0, "NodeID": 32, "Params0": 61, "Params1": 14, "Params2": 0, "Params3": 5, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}'

【问题讨论】：

不要使用单个 subprocess.run() 调用来运行所有内容，而是将命令写入脚本文件并执行 /bin/sh {filename}
嗨，我在&' 收到SyntaxError: invalid syntax，你能工作吗？
@VictorLee 这个错误是由于 test.text 文件中的新行字符

标签： python shell unix curl

【解决方案1】：

我最终使用了@barmar 的建议，即创建一个.sh 文件并通过/bin/sh 运行它。

【讨论】：

【解决方案2】：

尝试改变这个：

    while i <= int(vars(args)['e']):
        for line in f:
            cmd += f'curl -X POST -H "Content-Type: application/json" https://some_api_ep -d {line} &'
        i +=1
    subprocess.run(cmd, shell=True)

到：

    while i <= int(vars(args)['e']):
        for line in f:
            cmd = f'curl -X POST -H "Content-Type: application/json" https://some_api_ep -d {line} &'
            subprocess.run(cmd, shell=True)
        i +=1

或者你可以从这个“How to start a background process in Python?”查看

【讨论】：