Django 测试并行 AppRegistryNotReady答案

【问题标题】：Django test parallel AppRegistryNotReadyDjango 测试并行 AppRegistryNotReady
【发布时间】：2018-07-03 05:06:36
【问题描述】：

我正在尝试了解如何在内存 sqlite3 中并行运行 django 测试。

我有具有这种结构的 django 应用程序：

gbook
    order
        ...
        tests
            __init__.py
            test_a1.py
            test_b1.py
            utils.py

test_a1.py 和 test_b1.py 包含相同的代码：

import time
from order import models
from .utils import BackendTestCase


class ATestCase(BackendTestCase):
    def test_a(self):
        time.sleep(1)
        a = models.City.objects.count()
        self.assertEqual(a, a)

class BTestCase(BackendTestCase):
    def test_b(self):
        time.sleep(1)
        a = models.City.objects.count()
        self.assertEqual(a, a)

utils.py 是：

from django.test import TestCase, Client
from order import models
from django.conf import settings

from order.utils import to_hash


class BackendTestCase(TestCase):
    fixtures = ['City.json', 'Agency.json']

    def setUp(self):
        self.client = Client()
        self.lang_codes = (i[0] for i in settings.LANGUAGES)
        ...

settings_test.py：

from .settings import *

DEBUG = False

TEMPLATE_DEBUG = False

STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'

PASSWORD_HASHERS = ['django.contrib.auth.hashers.MD5PasswordHasher',]  # faster

DATABASES['default'] = {
    'ENGINE': 'django.db.backends.sqlite3',
}

当我在单个进程中运行测试时，一切顺利（大约 4 秒）：

python.exe manage.py test order --settings=gbook.settings_test

然后我尝试并行运行测试：

python.exe manage.py test order --settings=gbook.settings_test --parallel=2

我得到这个跟踪（控制台）：

Creating test database for alias 'default'...

Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (0 silenced).
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-1:
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\python\Python36-32\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\python\Python36-32\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\python\Python36-32\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\python\Python36-32\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\python\Python36-32\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\python\Python36-32\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\python\Python36-32\lib\multiprocessing\queues.py", line 337, in get
    return _ForkingPickler.loads(res)
  File "C:\python\Python36-32\lib\multiprocessing\queues.py", line 337, in get
    return _ForkingPickler.loads(res)
  File "C:\kvk\develop\Python\gbook\order\tests\test_a1.py", line 2, in <module>
    from order import models
  File "C:\kvk\develop\Python\gbook\order\tests\test_a1.py", line 2, in <module>
    from order import models
  File "C:\kvk\develop\Python\gbook\order\models.py", line 79, in <module>
    class Agency(models.Model):
  File "C:\kvk\develop\Python\gbook\order\models.py", line 79, in <module>
    class Agency(models.Model):
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 110, in __new__
    app_config = apps.get_containing_app_config(module)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 110, in __new__
    app_config = apps.get_containing_app_config(module)
  File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 247, in get_containing_app_config
    self.check_apps_ready()
  File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 247, in get_containing_app_config
    self.check_apps_ready()
  File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 125, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
  File "C:\python\venv\gbook\lib\site-packages\django\apps\registry.py", line 125, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

来自 Pycharm 的踪迹不同：

...

Traceback (most recent call last):
  File "C:\python\Python36-32\lib\unittest\suite.py", line 163, in _handleClassSetUp
    setUpClass()
  File "C:\python\venv\gbook\lib\site-packages\django\test\testcases.py", line 1036, in setUpClass
    'database': db_name,
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "C:\python\venv\gbook\lib\site-packages\modeltranslation\management\commands\loaddata.py", line 61, in handle
    return super(Command, self).handle(*fixture_labels, **options)
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 69, in handle
    self.loaddata(fixture_labels)
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 109, in loaddata
    self.load_label(fixture_label)
  File "C:\python\venv\gbook\lib\site-packages\django\core\management\commands\loaddata.py", line 175, in load_label
    obj.save(using=self.using)
  File "C:\python\venv\gbook\lib\site-packages\django\core\serializers\base.py", line 205, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 905, in _save_table
    forced_update)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\query.py", line 664, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 899, in execute_sql
    raise original_exception
  File "C:\python\venv\gbook\lib\site-packages\django\db\models\sql\compiler.py", line 889, in execute_sql
    cursor.execute(sql, params)
  File "C:\python\venv\gbook\lib\site-packages\django\db\backends\utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "C:\python\venv\gbook\lib\site-packages\django\db\utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "C:\python\venv\gbook\lib\site-packages\django\utils\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\python\venv\gbook\lib\site-packages\django\db\backends\utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "C:\python\venv\gbook\lib\site-packages\django\db\backends\sqlite3\base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: Problem installing fixture 'C:\kvk\develop\Python\gbook\order\fixtures\AirportInfo.json': Could not load order.AirportInfo(pk=2411): no such table: GB_AIRPORT_INFO

迁移似乎不适用于并行，但为什么呢？

文档说：“--parallel”在单独的并行进程中运行测试。每个进程都有自己的数据库。而且我不需要更改我的代码即可使用它。

请帮助我理解，我做错了什么。

multiprocessing.cpu_count() = 4

Django 版本 1.11.10

Python 3.6.5

【问题讨论】：

标签： python django testing parallel-processing

【解决方案1】：

MacOS 和 Python 3.8+ 的问题与上述相同。您必须在 settings.py 文件的顶部显式设置 import multiprocessing; multiprocessing.set_start_method('fork')。但请务必先了解副作用！

【讨论】：

谢谢！这实际上解决了我的问题（在 MacO 上）。如果您将使用它的后果包括在内，您的答案会好得多。也很容易只为测试启用它:)

【解决方案2】：

我在尝试在 Windows 上使用 --parallel 功能时遇到了类似的问题。

Django 的documentation 状态

此功能在 Windows 上不可用。它也不适用于 Oracle 数据库后端。

在 Linux 上运行相同的命令没有问题。

【讨论】：

我猜这是因为 fork 在 Windows 上不可用，所以多处理使用 spawn 代替它不擅长复制全局状态

【解决方案3】：

截至今天，Windows 上的并行运行仍处于禁用状态。您可以在此处跟踪保持此功能进展的工单：https://code.djangoproject.com/ticket/31169。

这是在 Windows 上禁用此选项的代码块：

def default_test_processes():
    """Default number of test processes when using the --parallel option."""
    # The current implementation of the parallel test runner requires
    # multiprocessing to start subprocesses with fork().
    if multiprocessing.get_start_method() != 'fork':
        return 1
    try:
        return int(os.environ['DJANGO_TEST_PROCESSES'])
    except KeyError:
        return multiprocessing.cpu_count()

来源：https://github.com/django/django/blob/59b4e99dd00b9c36d56055b889f96885995e4240/django/test/runner.py#L286-L295

【讨论】：

【解决方案4】：

回复@Menth，这就是我启用仅用于测试的方式：

# near the top of settings.py
if "test" in sys.argv[1:]:
    import multiprocessing

    logging.info("Using multiproc for testing.")
    multiprocessing.set_start_method("fork")

【讨论】：