剧作家的废壳答案

【问题标题】：Scrapy shell with playwright剧作家的废壳
【发布时间】：2022-07-20 18:00:23
【问题描述】：

是否可以在 Scrapy shell 中调用 Playwright？

我想使用一个 shell 来测试我的 xpath，我打算将它放在一个包含 Scrapy Playwright 的蜘蛛中。

我的 scrapy 设置文件具有通常的 Playwright 设置：

# Scrapy Playwright Setup
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

【问题讨论】：

标签： web-scraping scrapy playwright scrapy-shell

【解决方案1】：

我相信shell命令可能无法与scrapy playwright有关。

此文档链接应该可以进一步帮助您： https://playwright.dev/python/docs/intro#interactive-mode-repl

我相信你只需要交互模式下的 python3 或 python3 而不是 shell。这样你就有了scrapy shell从未做过的自动完成。

这是一个名为 spider_interactive.py 的文件中的同步示例：

from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()
browser = playwright.firefox.launch()
page = browser.new_page()
page.goto("http://whatsmyuseragent.org/")

#Remember to run these manually when your done to prevent left over garbage on the machine.
#browser.close()
#playwright.stop()

运行：

python3 -i spider_interactive.py

然后您可以输入例如以下命令：

page.locator("p.intro-text").all_inner_texts()

【讨论】：