【问题标题】:Strange output for threaded subprocesses线程子进程的奇怪输出
【发布时间】:2014-04-15 07:37:03
【问题描述】:

感谢您抽出宝贵时间查看此内容,我们将不胜感激任何帮助!

我正在尝试通过 ping 一个名为 ips 的 IP 地址列表来获取一些网络统计信息。然而,我遇到的问题是我的输出是一个包含几个“无”的列表。在实现线程运行子进程命令之前,下面显示的输出是一系列数字。如果有人可以看看我的源代码并阐明这个问题,我将非常感激!

提前感谢您!

import subprocess
import re
import threading
from multiprocessing import Pool, Lock
from multiprocessing.dummy import Pool as ThreadPool



def get_ips():

    # Fill empty list with IP address
    ips = []
    with open('C:\Python26\ARPips.prn','r')as f:
        for line in f:
            line = line[:-1]
            if line != "end":
                ips.append(line)
        return ips

def ping(pingArgs):


    lock = Lock()
    lock.acquire()

    # Ping with "pingArgs" as the arguments
    ping = subprocess.Popen(pingArgs,
        stdout = subprocess.PIPE,
        stderr = subprocess.PIPE,
        shell=True)

    # Get and parse output
    out = ping.communicate()
    out = ''.join((out))

    lost = re.findall(r"Lost = (\d+)", out)
    minimum = re.findall(r"Minimum = (\d+)", out)
    maximum = re.findall(r"Maximum = (\d+)", out)
    avg = re.findall(r"Average = (\d+)", out)
    no =  re.findall(r"Sent = (\d+)", out)

    # Change output to integers
    lost = [int(x) for x in lost]
    minimum = [int(x) for x in minimum]
    maximum = [int(x) for x in maximum]
    avg = [int(x) for x in avg]
    no = [int(x) for x in no]



    print "%s \t \t %s \t \t%s \t \t %s \t \t%s" % (no, lost, maximum, minimum, avg)
    lock.release()

def main():


    # grab IP address list
    ips = get_ips()

    # Declare global variables
    global position, newIP, pingArgs

    position = 0
    newIP = ips[position]
    position += 1
    pingArgs = ["ping", "-n", "1", "-l", "1", "-w", "100", newIP]


    # Header for output
    print "Packets \t loss(%) \t Max(ms) \t Min(ms) \t Average(ms)"

    # Instantiate Pool objects, and set size of pool
    pool = Pool(processes=12)


    #Ping ips in own threads and return the results
    result = pool.map(ping, ips)


    # Close the pool and wait for work to finish
    pool.close()
    pool.join()

    # print the results
    print result

if __name__ == '__main__':
    main()
    print get_ips()

输出如下所示:

Packets      loss(%)     Max(ms)     Min(ms)     Average(ms)
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[]       []         []       []         []
[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]
['10.10.10.1', '10.10.10.41', '10.10.10.42', '10.10.10.43', '10.10.10.49', '10.10.10.51', '10.10.10.61', '10.10.10.71', '10.10.10.91', '10.10.10.92', '10.10.10.201', '10.10.10.205', '10.10.10.208', '10.10.10.209', '10.10.10.213', '10.10.10.214']

Process finished with exit code 0

【问题讨论】:

  • 恕我直言锁should be global
  • 感谢您的回复,我已经附加了我的脚本。对输出有什么想法吗? :)
  • 看看out之后communicate()是否不为空。如果是,则缓冲可能存在问题。 Here 是一些细节。

标签: python multithreading parsing subprocess output


【解决方案1】:

我遇到的问题是我的输出是一个包含几个“无”的列表

Nones 由print result 产生,其中result = pool.map(ping, ips) 因为ping() 函数不返回任何内容(这意味着它在Python 中返回None)。

在实现线程运行子进程命令之前,如下所示的输出是一串数字。

您将 IP 地址传递给 Popen(),而不是完整的 ping 命令来运行。

您正在定义在任何地方都不使用的global pingArgsping() 中的本地pingArgs 掩盖了它)。您可以将pool.map 替换为在同一进程和同一线程中创建结果列表的内置map 调用,以查看结果是否相同(正则表达式不匹配)并且错误与线程无关/进程。

本地 lock = Lock() 没用。

这是修复上述问题的代码:

#!/usr/bin/env python
import re
from multiprocessing.dummy import Pool # use threads
from subprocess import Popen, PIPE

def get_ips(filename):
    ips = []
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if line and line != "end":
                ips.append(line)
    return ips

def ping(ip):
    cmd = ["ping", "-n", "1", "-l", "1", "-w", "100", ip]
    ping = Popen(cmd, stdout=PIPE, stderr=PIPE)

    # Get and parse output
    output, err = ping.communicate()
    out = ''.join([output, err])

    lost = re.findall(r"Lost = (\d+)", out)
    minimum = re.findall(r"Minimum = (\d+)", out)
    maximum = re.findall(r"Maximum = (\d+)", out)
    avg = re.findall(r"Average = (\d+)", out)
    no =  re.findall(r"Sent = (\d+)", out)

    # Change output to integers
    lost = [int(x) for x in lost]
    minimum = [int(x) for x in minimum]
    maximum = [int(x) for x in maximum]
    avg = [int(x) for x in avg]
    no = [int(x) for x in no]

    return "%s \t \t %s \t \t%s \t \t %s \t \t%s" % (
        no, lost, maximum, minimum, avg)

def main():
    # grab IP address list
    ips = get_ips(r'C:\Python26\ARPips.prn')

    # Header for output
    print "Packets \t loss(%) \t Max(ms) \t Min(ms) \t Average(ms)"

    # Instantiate Pool objects, and set size of pool
    pool = Pool(processes=12)

    #Ping ips in own threads and return the results
    results = pool.map(ping, ips)

    # Close the pool and wait for work to finish
    pool.close()
    pool.join()

    # print the results
    print "\n".join(results)

if __name__ == '__main__':
    main()

【讨论】:

  • 非常感谢您的详细解答!完美!
【解决方案2】:

这就是它为我(unix)系统工作的原因

import subprocess
import re
from multiprocessing import Pool, Lock

lock = Lock()
pingArgs = "ping -n -l 1 -w 100 "

def get_ips():
    # Fill empty list with IP address
    ips = []
    with open('C:\\Python26\\ARPips.prn', 'r')as f:
        for line in f:
            line = line[:-1]
            if line != "end":
                ips.append(line)
        return ips

def ping(ip):

    global lock
    pingCommand = pingArgs + ip
    lock.acquire()
    # Ping with "pingArgs" as the arguments
    ping = subprocess.Popen(pingCommand,
        stdout = subprocess.PIPE,
        stderr = subprocess.PIPE,
        shell=True)

    # Get and parse output
    out = ping.communicate()
    out = ''.join((out))

    lost = re.findall(r"Lost = (\d+)", out)
    minimum = re.findall(r"Minimum = (\d+)", out)
    maximum = re.findall(r"Maximum = (\d+)", out)
    avg = re.findall(r"Average = (\d+)", out)
    no =  re.findall(r"Sent = (\d+)", out)

    # Change output to integers
    lost = [int(x) for x in lost]
    minimum = [int(x) for x in minimum]
    maximum = [int(x) for x in maximum]
    avg = [int(x) for x in avg]
    no = [int(x) for x in no]

    lock.release()
    return "%s \t \t %s \t \t%s \t \t %s \t \t%s" % (no, lost, maximum, minimum, avg)

def main():

    # grab IP address list
    ips = get_ips()

    # Header for output
    print("Packets \t loss(%) \t Max(ms) \t Min(ms) \t Average(ms)")

    # Instantiate Pool objects, and set size of pool
    pool = Pool(processes=12)

    #Ping ips in own threads and return the results
    results = pool.map(ping, ips)

    # Close the pool and wait for work to finish

    pool.close()
    pool.join()
    # print the results
    for result in results:
        print(result)

if __name__ == '__main__':
    main()
    print(get_ips())

我在代码中遇到的问题是不正确的 ping 参数(-n 没有带任何其他参数),lock 应该是全局的,当 shell=True 没有带列表时,只有一个字符串,而ping(pingArgs) 可以覆盖全局参数,而您希望池仅将 ip 发送给工作人员,然后工作人员应将其添加为最终参数。

我还注意到您没有转义到 dist 文件的路径,我没有使用/测试该部分,但也将其正确转义以供参考。

编辑:还对其进行了更改,因此每个函数都将简单地返回要打印的值并在完成时打印所有这些值。而不是将它们打印在函数内部以及没有返回的函数结果(无返回值列表)。

【讨论】:

  • 非常感谢您的回答!由于我不在 linux 系统上工作,我只测试了第一次工作的另一个答案。为了其他人的利益,我不得不将另一个答案标记为正确,但是如果我能给出两个正确的答案,我会的!再次感谢!
  • 很高兴为您提供帮助,很高兴您知道了!
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 2020-12-29
  • 1970-01-01
  • 2021-07-03
  • 1970-01-01
  • 1970-01-01
  • 2013-03-15
  • 2011-03-18
相关资源
最近更新 更多