【问题标题】:Trouble Installing Scrapy on 10.9 Mavericks在 10.9 Mavericks 上安装 Scrapy 时遇到问题
【发布时间】:2014-04-02 03:15:12
【问题描述】:

我正在尝试使用我的自制 python 版本安装 Scrapy。尽管所有内容都是最新的,已安装并在我的路径中,但我无法使用它。

这是我的操作顺序:

检查酿造:

$ brew doctor
Your system is ready to brew.

检查 python 和 pip:

$ which python
/usr/local/bin/python
$ python --version
Python 2.7.6
$ which pip
/usr/local/bin/pip
$ pip -- version
pip 1.5.4 from /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg (python 2.7)

(这正常吗?好像有两个副本,可能有冲突)

我已经安装了scrapy,但是当我再次尝试时,我得到:

$ pip install scrapy
Requirement already satisfied (use --upgrade to upgrade): scrapy in /usr/local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->scrapy)
Cleaning up...

这对我来说很奇怪,因为我希望我的模块安装在 /usr/local/share/python 而不是 /usr/local/lib/python2.7/site-packages 果然,/usr/local/lib/python2.7/site-packages 没有scrapy,但/usr/local/share/python 有。

我已将/usr/local/share/python/scrapy添加到我的路径中,如下所述:Scrapy installation on OSX Lion

which scrapy 返回空白。

最有趣的部分是当我尝试使用pip install --upgrade scrapy“更新”scrapy 时:

~ ∮ pip install --upgrade scrapy
Requirement already up-to-date: scrapy in /usr/local/lib/python2.7/site-packages
Requirement already up-to-date: Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Downloading/unpacking lxml from https://pypi.python.org/packages/source/l/lxml/lxml-3.3.2.tar.gz#md5=a3ea7bf74b718ecb46d9fd5198eec92d (from scrapy)
  Downloading lxml-3.3.2.tar.gz (3.5MB): 3.5MB downloaded
  Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/lxml/setup.py) egg_info for package lxml
    /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    Building lxml version 3.3.2.
    Building without Cython.
    Using build configuration of libxslt 1.1.28

    warning: no previously-included files found matching '*.py'
Downloading/unpacking pyOpenSSL from https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.14.tar.gz#md5=8579ff3a1d858858acfba5f046a4ddf7 (from scrapy)
  Downloading pyOpenSSL-0.14.tar.gz (128kB): 128kB downloaded
  Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/pyOpenSSL/setup.py) egg_info for package pyOpenSSL

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    no previously-included directories found matching 'doc/_build'
Requirement already up-to-date: cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already up-to-date: zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Downloading/unpacking cryptography>=0.2.1 (from pyOpenSSL->scrapy)
  Downloading cryptography-0.2.1.tar.gz (13.8MB): 13.8MB downloaded
  Running setup.py (path:/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py) egg_info for package cryptography
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found

    Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/cffi-0.8.1-py2.7-macosx-10.9-x86_64.egg
    Searching for pycparser
    Reading https://pypi.python.org/simple/pycparser/
    Best match: pycparser 2.10
    Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.10.tar.gz#md5=d87aed98c8a9f386aa56d365fe4d515f
    Processing pycparser-2.10.tar.gz
    Writing /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/setup.cfg
    Running pycparser-2.10/setup.py -q bdist_egg --dist-dir /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/egg-dist-tmp-pniw4C
    zip_safe flag not set; analyzing archive contents...

    Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/pycparser-2.10-py2.7.egg

    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 113, in <module>
        "build": cffi_build,
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
        dist.run_commands()
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "<string>", line 15, in replacement_run
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 186, in find_sources
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 246, in run
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 282, in add_defaults
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/sdist.py", line 167, in add_defaults
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/build_py.py", line 26, in finalize_options
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options
        ('force', 'force'))
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options
        src_cmd_obj.ensure_finalized()
      File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 52, in finalize_options
        from cryptography.hazmat.primitives import constant_time, padding
      File "cryptography/hazmat/primitives/constant_time.py", line 18, in <module>
        import six
    ImportError: No module named six
    Complete output from command python setup.py egg_info:
    Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found

Package libffi was not found in the pkg-config search path.

Perhaps you should add the directory containing `libffi.pc'

to the PKG_CONFIG_PATH environment variable

No package 'libffi' found



Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/cffi-0.8.1-py2.7-macosx-10.9-x86_64.egg

Searching for pycparser

Reading https://pypi.python.org/simple/pycparser/

Best match: pycparser 2.10

Downloading https://pypi.python.org/packages/source/p/pycparser/pycparser-2.10.tar.gz#md5=d87aed98c8a9f386aa56d365fe4d515f

Processing pycparser-2.10.tar.gz

Writing /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/setup.cfg

Running pycparser-2.10/setup.py -q bdist_egg --dist-dir /var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/easy_install-qqW3QR/pycparser-2.10/egg-dist-tmp-pniw4C

zip_safe flag not set; analyzing archive contents...



Installed /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/pycparser-2.10-py2.7.egg

running egg_info

creating pip-egg-info/cryptography.egg-info

writing requirements to pip-egg-info/cryptography.egg-info/requires.txt

writing pip-egg-info/cryptography.egg-info/PKG-INFO

writing top-level names to pip-egg-info/cryptography.egg-info/top_level.txt

writing dependency_links to pip-egg-info/cryptography.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/cryptography.egg-info/SOURCES.txt'

warning: manifest_maker: standard file '-c' not found



Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 113, in <module>

    "build": cffi_build,

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup

    dist.run_commands()

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands

    self.run_command(cmd)

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command

    cmd_obj.run()

  File "<string>", line 15, in replacement_run

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 186, in find_sources

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 246, in run

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/egg_info.py", line 282, in add_defaults

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/sdist.py", line 167, in add_defaults

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command

    cmd_obj.ensure_finalized()

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

    self.finalize_options()

  File "build/bdist.macosx-10.9-x86_64/egg/setuptools/command/build_py.py", line 26, in finalize_options

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options

    ('force', 'force'))

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options

    src_cmd_obj.ensure_finalized()

  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

    self.finalize_options()

  File "/private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_pauluslabuser/cryptography/setup.py", line 52, in finalize_options

    from cryptography.hazmat.primitives import constant_time, padding

  File "cryptography/hazmat/primitives/constant_time.py", line 18, in <module>

    import six

ImportError: No module named six

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /private/var/folders/22/6lh7b3m53zx1xslwtkdk5v400000gp/T/pip_build_myAccount/cryptography
Storing debug log for failure in /Users/myAccount/.pip/pip.log
~ ∮ 

我尝试过修复其中的一些问题,例如pip install six,并尝试在谷歌上搜索此egg_info failed with error code 1。 大多数解决方案都涉及尚未安装 setuptools:

$ pip install setuptools
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg
Cleaning up...

它看起来像我的 python brew 安装和不正确的链接/多个版本? pip 导致模块安装在一个地方,而我的 python 在其他地方(这解释了为什么 &gt;&gt;&gt; import scrapy 失败但不能解释为什么 $ scrapy 失败)。

关于损坏的 python 模块的类似帖子建议运行xcode-select --install,我已经确认它是最新的。 Can't install PIL after Mac OS X 10.9

也许它的 virtualenv 时间?

提前致谢!

【问题讨论】:

  • 是的,virtualenv 可能会解决很多这些问题
  • hmmm...我已经在不使用自制软件的情况下完成了此操作...只需执行sudo pip install scrapy(sudo 因为通常 mac 权限被搞砸了),它应该工作......除非你的电脑有问题。
  • @aj8uppal 我得到的结果与使用pip install scrapy 时相同在命令行不识别,python 也不识别&gt;&gt;&gt; import scrapy
  • @hd1 我已经尝试将virtualenv 安装到与scrapy 大致相同的效果。我关注了:stackoverflow.com/questions/16860971/… 关注了第一个答案并且:~ ∮ pip install virtualenv Requirement already satisfied (use --upgrade to upgrade): virtualenv in /usr/local/lib/python2.7/site-packages Cleaning up... ~ ∮ virtualenv test -bash: virtualenv: command not found
  • 执行时是否使用scrapy的完整路径?如果没有,请尝试。

标签: python macos python-2.7 scrapy osx-mavericks


【解决方案1】:

我在安装virtualenv:Can't pip install virtualenv on Mac 10.9 Mavericks with brewed python时遇到了类似的问题

那里的解决方案在这里也有效,尽管我仍然不明白为什么首先会出现问题以及为什么这个解决方案有效。但确实如此。

卸载并重新安装scrapy:

$pip uninstall scrapy
$pip install scrapy 

这会从 /usr/local/lib/python2.7/site-packages 中删除模块 并将它们安装在/usr/local/bin/

问题在于,根据自制 python 文档,pip 应该安装在/site-packages 中,它们应该可以工作。 它不仅不适用于/site-packages 中的它们,而且 pip 默认安装在/usr/local/bin/ 中。

此解决方法有效,但我非常感谢您解释它为什么有效以及为什么这是一个问题。那么也许这两个问题可以合并?

【讨论】:

    【解决方案2】:

    这对我有用:

    1. 由于您使用 Homebrew 作为包管理器,因此通过它安装的程序将设置为 /usr/local/bin,而不是开箱即用应用程序的标准 /usr/bin 路径设置。您需要确保/usr/local/bin 位于/etc/paths 文件的顶部。该文件应该被锁定,因此您可能需要在 vi 编辑器中使用以下内容保存更改:

      :w !sudo tee %
      
    2. $ which python 应该返回 /usr/local/bin/python,否则:

      `$ brew unlink python`
      `$ brew install python`
      `$ brew linkapps`
      
    3. $ which pip 应该返回 /usr/local/bin/pip,否则

      `$ brew install pip`
      
    4. 那么……

      `$ sudo pip uninstall Scrapy`
      `$ pip install Scrapy`
      `$ pip install --upgrade Scrapy`
      `$ pip install --upgrade lxml`
      `$ scrapy startproject tutorial`
      

    【讨论】:

      【解决方案3】:

      您需要让您的安装环境访问与 OpenSSL 捆绑在一起的 Crypto 库。这是对我有用的解决方案。

      env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include"

      请记住将其更改为更新的 OpenSSL 包的位置。

      【讨论】:

        【解决方案4】:

        其他帖子对我不起作用,但确实如此:

        sudo ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install scrapy
        

        【讨论】:

          【解决方案5】:

          以上方法在 OS X 10.10 Yosemite 中都不适合我。

          但我找到了解决方案here

          它正在使用 pkg-config:

          brew install pkg-config libffi
          export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.0.13/lib/pkgconfig/
          pip install cffi
          

          您必须根据您的 libffi 版本更改路径。

          【讨论】:

          • 这在 10.9 Mavericks 中也适用于我。使用locate libffi.pc 找到我的:/usr/local/Cellar/libffi/3.0.13/lib/pkgconfig/libffi.pc
          猜你喜欢
          • 2013-11-05
          • 2013-11-12
          • 1970-01-01
          • 1970-01-01
          • 1970-01-01
          • 2013-11-26
          • 2014-05-26
          • 1970-01-01
          • 1970-01-01
          相关资源
          最近更新 更多