【问题标题】:Perdiodical data fetch should block other threads using a schedueler定期数据获取应该使用调度程序阻塞其他线程
【发布时间】:2017-11-02 15:22:04
【问题描述】:

我是 python 的新手。我很感激任何帮助。那么让我们开始吧。

我正在运行一个 FLASK 应用程序作为休息 api。对于每个请求,都会返回一个 json。每秒可能有大约 20 个请求。在后台,我使用 APscheduler 每 60 秒从 ldap 获取实际数据。

scheduler = BackgroundScheduler()
scheduler.add_job(
    func=my_ldap.fetch_people_ldap,
    trigger=IntervalTrigger(seconds=60),
    id='fetching_data_job',
    name='Fetch data from ldap every 60 seconds')

scheduler.start()
atexit.register(lambda : scheduler.shutdown())

但实际上,当使用 api 调用访问数据时,应用程序正在关闭内存问题的原因,我认为这是我在访问 ldap 对象的同时 fetch_people_ldap- 函数被调用的原因调度器。

我想通过阻塞处理 api 调用的线程直到 ldap 数据获取成功退出来解决这个可怕的错误。但我不知道该怎么做。

有什么建议或解决办法吗?

这是我得到的错误日志: 致命的 Python 错误:两次保存线程?

Thread 0x00007f95f3fff700 (most recent call first):
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 294 in _ldap_call
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 721 in result4
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 714 in result3
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 707 in result2
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 703 in result
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 796 in search_ext_s
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 802 in search_s
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 49 in fetch_people_ldap
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/apscheduler/executors/base.py", line 125 in run_job
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55 in run
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 66 in _worker
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x00007f95f8910700 (most recent call first):
  File "/usr/lib/python3.5/threading.py", line 297 in wait
  File "/usr/lib/python3.5/threading.py", line 549 in wait
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/apscheduler/schedulers/blocking.py", line 28 in _main_loop
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Current thread 0x00007f9605505700 (most recent call first):
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 294 in _ldap_call
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 791 in search_ext
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 795 in search_ext_s
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 802 in search_s
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 59 in check_node
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 118 in build_tree_recursive
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 124 in build_tree_recursive
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 124 in build_tree_recursive
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 129 in build_tree
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/app/app.py", line 83 in get_trainings_by_unit
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 1598 in dispatch_request
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 1612 in full_dispatch_request
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 1982 in wsgi_app
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 1997 in __call__
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 197 in execute
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 209 in run_wsgi
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 267 in handle_one_request
  File "/usr/lib/python3.5/http/server.py", line 422 in handle
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 232 in handle
  File "/usr/lib/python3.5/socketserver.py", line 681 in __init__
  File "/usr/lib/python3.5/socketserver.py", line 354 in finish_request
  File "/usr/lib/python3.5/socketserver.py", line 341 in process_request
  File "/usr/lib/python3.5/socketserver.py", line 313 in _handle_request_noblock
  File "/usr/lib/python3.5/socketserver.py", line 234 in serve_forever
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 539 in serve_forever
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 702 in inner
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 739 in run_simple
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 841 in run
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/app/app.py", line 92 in <module>

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

【问题讨论】:

    标签: python multithreading scheduler apscheduler


    【解决方案1】:

    问题是您的 API 调用处理程序也在调用 LDAP 以创建响应。如果您要每秒收到 20 个请求,那么这将导致问题。预定的通话也没有多大意义,因为您何时使用检索到的数据?

    我的方法是让计划任务在后台更新公共数据存储。然后,您的 API 调用仅从该公共数据存储读取数据,而根本不必触及 LDAP。您使用数据库还是仅使用内存取决于您正在处理的数据的大小和复杂性。

    主要是将创建 API 响应与从 LDAP 获取数据分离。

    编辑: 我从您的评论中了解到,您认为您的 API 仅使用您自己的 ldapCheck.py 代码,但如果您查看完整的堆栈跟踪,它实际上是使用 LDAP 库进行调用:

    API 线程:

    Current thread 0x00007f9605505700 (most recent call first):
      File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 294 in _ldap_call
    ...
      File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 124 in build_tree_recursive
    ...
    File "/path/to/folder/tt_report_api/training_tool_report_api/sample/app/app.py", line 83 in get_trainings_by_unit
    ...
      File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 841 in run
    

    当这种情况发生在您计划的后台线程调用 LDAP 的同时,您会收到此错误。

    Thread 0x00007f95f3fff700 (most recent call first):
      File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 294 in _ldap_call
    ...
      File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 49 in fetch_people_ldap
    ...
      File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap
    

    解决方案是不要阻塞您的 API 线程。解决方案是确保ldapCheck.py", line 59 in check_node 中的代码不会调用您的 LDAP 库,而是使用后台线程已检索和存储的信息。

    【讨论】:

    • 嗨,巴特,感谢您的回复。我认为您描述的方法是我在工具中使用的方法。我有一个本地 ldap 对象,并在间隔后再次获取数据,以使数据保持最新。由于我使用的是 Apscheduler,问题是使用 ldap 对象和调度程序的功能想要同时访问数据,导致它们在多个线程中运行。有没有办法在子线程从 ldap 获取数据时暂停主线程?
    猜你喜欢
    • 2013-07-11
    • 1970-01-01
    • 1970-01-01
    • 2019-11-21
    • 2016-03-27
    • 2012-11-08
    • 1970-01-01
    • 1970-01-01
    • 2014-04-03
    相关资源
    最近更新 更多