如何在单台机器上运行具有多处理功能的 Locust答案

【问题标题】：How to run Locust with multiprocessing on a single machine如何在单台机器上运行具有多处理功能的 Locust
【发布时间】：2020-08-07 12:37:38
【问题描述】：

我希望 Locust 使用我 PC 上的所有内核。

我有很多 Locust 类，我想将 Locust 用作库。

我的代码示例：

import gevent
from locust.env import Environment
from locust.stats import stats_printer
from locust.log import setup_logging
import time



from locust import HttpUser, TaskSet, task, between


def index(l):
    l.client.get("/")

def stats(l):
    l.client.get("/stats/requests")

class UserTasks(TaskSet):
    # one can specify tasks like this
    tasks = [index, stats]

    # but it might be convenient to use the @task decorator
    @task
    def page404(self):
        self.client.get("/does_not_exist")

class WebsiteUser(HttpUser):
    """
    User class that does requests to the locust web server running on localhost
    """
    host = "http://127.0.0.1:8089"
    wait_time = between(2, 5)
    tasks = [UserTasks]

def worker():
    env2 = Environment(user_classes=[WebsiteUser])
    env2.create_worker_runner(master_host="127.0.0.1", master_port=50013)
    # env2.runner.start(10, hatch_rate=1)
    env2.runner.greenlet.join()

def master():
    env1 = Environment(user_classes=[WebsiteUser])
    env1.create_master_runner(master_bind_host="127.0.0.1", master_bind_port=50013)
    env1.create_web_ui("127.0.0.1", 8089)
    env1.runner.start(20, hatch_rate=4)
    env1.runner.greenlet.join()

import multiprocessing
from multiprocessing import Process
import time


procs = []

proc = Process(target=master)
procs.append(proc)
proc.start()

time.sleep(5)

for i in range(multiprocessing.cpu_count()):
    proc = Process(target=worker)  # instantiating without any argument
    procs.append(proc)
    proc.start()

for process in procs:
    process.join()

此代码无法正常工作。

(env) ➜  test_locust python main3.py
You are running in distributed mode but have no worker servers connected. Please connect workers prior to swarming.
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/alex/projects/performance/env/lib/python3.6/site-packages/locust/runners.py", line 532, in client_listener
    client_id, msg = self.server.recv_from_client()
  File "/home/alex/projects/performance/env/lib/python3.6/site-packages/locust/rpc/zmqrpc.py", line 44, in recv_from_client
    msg = Message.unserialize(data[1])
  File "/home/alex/projects/performance/env/lib/python3.6/site-packages/locust/rpc/protocol.py", line 18, in unserialize
    msg = cls(*msgpack.loads(data, raw=False, strict_map_key=False))
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'
2020-08-13T11:21:10Z <Greenlet at 0x7f8cf300c848: <bound method MasterRunner.client_listener of <locust.runners.MasterRunner object at 0x7f8cf2f531d0>>> failed with TypeError

Unhandled exception in greenlet: <Greenlet at 0x7f8cf300c848: <bound method MasterRunner.client_listener of <locust.runners.MasterRunner object at 0x7f8cf2f531d0>>>
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/alex/projects/performance/env/lib/python3.6/site-packages/locust/runners.py", line 532, in client_listener
    client_id, msg = self.server.recv_from_client()
  File "/home/alex/projects/performance/env/lib/python3.6/site-packages/locust/rpc/zmqrpc.py", line 44, in recv_from_client
    msg = Message.unserialize(data[1])
  File "/home/alex/projects/performance/env/lib/python3.6/site-packages/locust/rpc/protocol.py", line 18, in unserialize
    msg = cls(*msgpack.loads(data, raw=False, strict_map_key=False))
  File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
TypeError: unpackb() got an unexpected keyword argument 'strict_map_key'

实际结果：worker 不会连接到 master 并在没有 master 的情况下运行用户预期结果：worker 只与 master 一起运行。

怎么了？

【问题讨论】：

感谢您的提问！您能否详细说明运行代码时所需的结果是什么，以及会发生什么？请尽可能编辑原始问题。
太好了，感谢您的编辑！您能否详细说明一下您是如何运行代码的，以及为什么需要使用 python-multiprocessing ？如果我理解正确，运行多个 Locust 工作人员的首选方法是为每个工作人员 use a separate OS process 以充分利用 CPU。不过，this question 表明您的方法也是可行的。
我知道如何使用 bash 脚本以分布式模式运行 locust，但我的目标是使用 locust 作为库。
我需要使用 CPU 的所有内核，因为我有一个高负载性能应用程序。

标签： python python-multiprocessing locust

【解决方案1】：

您不能将 multiprocessing 与 Locust/gevent 一起使用（或者至少已知会导致问题）。

请使用 subprocess 或完全在 locust 外部的东西生成单独的进程。或许您可以修改 locust-swarm (https://github.com/SvenskaSpel/locust-swarm) 使其能够在同一台机器上运行工作进程。

【讨论】：